by S. Donike & A. Porti (2021)
Introduction
Within the end of term assignment for the Practice of Software Development course, a KML file is created visualizing both the result of an OGC Web Map Service1 request as well as the parsed information from a CSV file containing tweets and their locations. Additionally, the tweets are analysed regarding possible profanity and visualized accordingly.
The following report will detail the structure of the code used to implement such visualization and the design decisions carried out.
Implementation and Design
The general structure of the project submission is comprised of four Java classes:
- GoogleEarthTweetMapper.java
- WMSconnector.java
- CSVtoKML.java
- CSVtoKML_polygon.java
All java files, including the KML outputs, as well as the file with the tweets (tweets.csv) and other external used sources, can be found in a public GitHub Repository.
GoogleEarthTweetMapper
The GoogleEarthTweetMapper.java is the main class, which calls functions from the other classes and saves the resulting WMS image.
As shown in Figure 1 the WMSconnector is called first by instantiating a WMSconnector object. Then, on that object, the getWMSimage(…) method is called, which returns a Buffered Image (from now on BI). This is then saved and written to the Hard Drive via a file writer, passing the BI to the WMSconnector.save_image(…) function. Next, the CSVtoKML class is called, passing the directory and file name of the twitter.csv file, which saves the created KML into the working directory. The same is done with the CSVtoKML_polygon class, which receives the same arguments. The created KML is saved to the Hard Drive from within the class. It would also be possible to write the finished HTTP WMS request into the KML so that it is dynamically loaded when opening the file, but it was explicitly stated in the task to store the WMS PNG on the hard drive.
package eot_donike_porti;
import java.awt.image.BufferedImage;
public class GoogleEarthTweetMapper {
public static void main(String[] args) {
/* connect to WMSconnector class as wms_connection */
WMSconnector wms_connection = new WMSconnector();
/* get image as return from function getWMSimage in WMSconnector class */
BufferedImage buffered_image = wms_connection.getWMSimage();
/* pass image to saveImage function from WMSconnector class */
boolean b = wms_connection.save_image(buffered_image);
/* defining path of input files (twitter CSV, profanity TXT) */
String working_dir = "/Users/simondonike/Documents/GitHub/eot_donike_porti/out/production/final_assignment/eot_donike_porti/";
/* calling CSV to KML class & funtion to write point kml */
CSVtoKML.read_convert_save(working_dir, "twitter.csv");
/* calling CSV to KML class & funtion to write and style polygon kml */
CSVtoKML_polygon.read_convert_save_polygon(working_dir, "twitter.csv", "profanity_list.txt");
}
} WMSconnector Class
The WMSconnector.java sends the WMS request and returns a BI, which is then saved by the main class.
The WMSconnector class contains two functions: getWMSimage() and save_image(BI) (see Figure 2). The first function is called from main without any arguments, since all relevant parameters are hard-coded into GetMapRequest.request. Before that, a try-catch block validates that the input WMS URL is valid. After sending the request via .issueRequest(wms), the returning image is read via getInputStream() and saved as a BI variable and returned to main. From main, the BI is then sent to the second function, which is passed the BI and then saves it as PNG to the working directory.
package eot_donike_porti;
import org.geotools.ows.wms.WebMapServer;
import org.geotools.ows.wms.request.GetMapRequest;
import org.geotools.ows.wms.response.GetMapResponse;
import javax.imageio.ImageIO;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
public class WMSconnector {
public BufferedImage getWMSimage() {
WebMapServer wms = null;
URL wmsURL = null;
try {
wmsURL = new URL("http://maps.heigit.org/osm-wms/wms");
}
catch (MalformedURLException e) {
e.printStackTrace();
}
try {
//Creating a new WebMapServer
wms = new WebMapServer(wmsURL);
GetMapRequest request = wms.createGetMapRequest();
/* setting parameters of request, hardcoding request parameters :( */
request.setVersion("1.3.0");
request.setFormat("image/png");
request.setDimensions(2500, 2500);
request.setTransparent(true);
request.setSRS("EPSG:4326");
request.setBBox("42.32,-71.13,42.42,-71.03");
request.addLayer("osm_auto:all", "default");
/* send request */
GetMapResponse response = (GetMapResponse) wms.issueRequest(request);
/* reading response via input stream, saving as BI */
BufferedImage return_image = ImageIO.read(response.getInputStream()); // why error?
return return_image;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
/* function to save image, returns boolean if it worked or not */
public boolean save_image(BufferedImage img) {
/* save as File data type and set name */
File output_file = new File("output_WMS.png");
try {
/* write file set file type */
ImageIO.write(img,"png",output_file); //save file, file type png
return true; // return true boolean
} catch (IOException e) {
e.printStackTrace(); // print error
return false; // return false boolean
} // close catch block
} // close function
} // close class
CSVtoKML class
This java class takes the working directory and the CSV file name as arguments when it is called from main. As displayed in Figure 3, it only contains one function, which handles all operations. This was done because the writing of the KML is quite straight forward, the iterating over the CSV file can easily be done with a while loop and therefore it seems overly complicated to split the steps up in different functions. First, a ground overlay element is created via pre-defined strings and attached to the KML string. The transparency is set via the color tag, which is explicitly not intended for use with raster data, but it works. The WMS image is inserted by pointing to the correct storage location of the PNG (see Figure 4). Then, the CSV file is read via Buffered Reader and an empty String Array created. Also, the string which will later on contain the whole KML string is created and filled with the KML header information. Iterating over the tweets list (excluding the header) via a while loop, the information for each tweet is read and stored in the array. Using string concatenation, the relevant KML tags are opened and the info from the CSV line inserted. Figure 2. WMSconnector diagram. 3 The extended data tags are used because they are shown as a nicely formatted table when clicking on the icons in Google Earth Pro, as visible in Figure 5. Additionally, the timestamp from the tweet does not conform to the specifications as given by the KML documentation, so the space is replaced with a T and the hour added to the end of the string. The timestamps are saved within “TimeStamp”2 KML tags so that Google Earth can correctly identify the time series of the tweets. After each iteration, the string holding this line’s tweet information is appended to the KML string. All statements within that KML string are given with formatting statements such as tabs and new lines. At the end, the KML string s saved as a KML file and can be opened with Google Earth Pro.
package eot_donike_porti;
import java.io.*;
public class CSVtoKML {
public static void read_convert_save(String working_dir, String csv_fileName) {
try {
File file = new File(working_dir+csv_fileName);
FileReader fr = new FileReader(file);
BufferedReader br = new BufferedReader(fr);
String line = "";
String[] tempArr;
/* Create KML string with document header */
String kml = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
+ "<kml xmlns=\"http://www.opengis.net/kml/2.2\">\n"
+ "<Document>\n\n";
/* Adding png from WMS as overlay */
kml = kml + "<GroundOverlay>\n"
+ "\t<name>WMS Overlay</name>\n" // setting name
+ "\t<color>64ffffff</color>\n" // set opacity with hex
+ "\t<Icon> <href>output_WMS.png</href> </Icon>\n" // file path of png
+ "\t<LatLonBox>\n" // opening bbox tag
+ "\t\t<north>42.32</north>\n" // setting bb
+ "\t\t<south>42.42</south>\n" // setting bb
+ "\t\t<east>-71.03</east>\n" // setting bb
+ "\t\t<west>-71.13</west>\n" // setting bb
+ "\t</LatLonBox>\n" // closing bbox tag
+ "</GroundOverlay>\n"; // closing ground overlay tag
/* iterate over csv file by lines until line is empty */
while((line = br.readLine()) != null) {
/* create Array per line, separate based on semicolon */
tempArr = line.split(";");
/* check if temp array is != id, therefore excluding the csv header line */
if (!tempArr[0].equals("id")) {
/* fill temp string w/ kml placemark syntax + info from iterated array incl. indentation */
String temp_placemark = "<Placemark>\n" // start with opening placemark tag
//+ "\t<name>" + tempArr[0] + "</name>\n" // defining name of PM as ID of tweet
+ "\t<ExtendedData>\n" // open extended data tag
+ "\t\t<Data name=\"TweetID\"> " + "<value>" + tempArr[0] + "</value> </Data>\n" // sub-tag w/TweetID
+ "\t\t<Data name=\"Tweet\"> " + "<value>" + tempArr[5] + "</value> </Data>\n" // sub-tag w/Tweet
+ "\t\t<Data name=\"Hashtags\"> " + "<value>" + tempArr[3] + "</value> </Data>\n" // sub-tag w/Hashtags, only shows up in GE if hastags in tweet
+ "\t\t<Data name=\"TimeStamp\" >" + "<value>" + tempArr[6].replace(' ', 'T') + ":00" + "</value> </Data>\n" // sub-tag w/CreatedAt
+ "\t\t<Data name=\"UserID\"> " + "<value>" + tempArr[7] + "</value> </Data>\n" // sub-tag w/userID
+ "\t</ExtendedData>\n" // close extended data tag
/* magic to turn timestamp into dateTime format: (YYYY-MM-DDThh:mm:sszzzzzz) */
+ "\t<TimeStamp id=\"" + tempArr[0] + "\"> <when>" + tempArr[6].replace(' ', 'T') + ":00" + "</when> </TimeStamp>\n"
+ "\t<Point> <coordinates>" + tempArr[1] + "," + tempArr[2] + "</coordinates> </Point>\n" // Setting coordinates
+ "</Placemark>\n"; // closing placemark tag
/* append kml string with Placemark info */
kml = kml + temp_placemark;
} // close if statement
} // end while loop
/* close buffered Reader */
br.close();
/* append kml with document/kml closing tags after iteration */
kml = kml + "</Document>\n"+"</kml>";
/* Write KML */
FileWriter fw = new FileWriter("output.kml");
fw.write(kml);
fw.close();
/* end try block (reading csv), defining and closing catch block */
} catch(IOException ioe) { ioe.printStackTrace(); }
} // end reader
} // end CSVtoKML class
CSVtoKML_polygon Class
CSVtoKML_polygon.java performs the same iteration and KML creation as the CSVtoKML class, but additionally creates an extruded polygon (instead of points shown on the map) and checks the tweets for profanity, colorcoding the polygons to visualize whether or not the tweets contain profanity. Therefore the inner workings of the KML creation will not be detailed again, but are visualized in a diagram in Figure 6.
Since the creation and extrusion of polygons, as well as the saving of KML style templates and the opening, checking and visualization of the tweets according to their profanity content contains quite a lot more complexity, many of those tasks were outsourced to functions. The functions are all called from within the read_convert_save_polygon(…) function, therefore only one call from main is necessary to start the whole process.
First, after writing the opening KML tags to the string, the create_styles(…) function is called. This returns style templates for the polygons as a string which is written to the KML string and can later be referenced by their IDs from within the polygon tags.
Then, within every loop over a new line of the tweets.csv file, the create_profanity_color_string(…) function is called. This function is passed the tweet itself as well as the location and name of the profanity list file. For each tweet, the read_profanity_list(…) function is called, which receives the location and name of the profanity list and opens it, returning it as a String Array. Back in the create_profanity_color_string(…) function, this String Array4 is iterated over and checked for matches with the tweet. If a match is found, the KML color style tag for “red” is received and written into the KML structure of the polygon of said tweet. The detection is not perfect, since checking for the words itself would return many false positives (“I passed my exam”). Therefore, a space in front of each offensive word is added to exclude such false positives. Adding a space behind the word would have excluded many common expressions where a punctuation mark is added at the end, such as “you’re a bitch!”.
After the style information is added according to the profanity content, the polygon around the point is created. The create_polygon_string(…)function is called, which receives the coordinates of the tweet. Adhering to the KML polygon specifications5, the polygon tags are created as strings. The coordinates are transformed, adding and subtracting a certain value to and from the coordinate to create a square around the coordinate, as explained in the schema in Figure 7. Also, the parameters for extrusion are enabled and set as relative to ground, meaning the polygons are extruded from the surface by 100m (as specified with the z coordinate in the coordinate section). The KML string containing the polygon is then returned to the loop.
Since for each tweet, the information of the tweet itself, the polygon and the check for profanity and the definition of the according style is now done (see Figure 8), the temporary tweet KML string can be added to the evergrowing KML string. Finally, the KML string is stored as “output_polygon.kml” in the working directory.
package eot_donike_porti;
import java.io.*;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class CSVtoKML_polygon {
public static String[] read_profanity_list(String working_dir, String profanity_fileName) throws IOException {
/* reads list from file and returns as String List, passing down file & path */
BufferedReader reader = new BufferedReader(new FileReader(working_dir+profanity_fileName));
List<String> data_list = new ArrayList<String>();
String line_string;
while((line_string = reader.readLine())!=null) {
data_list.add(line_string);
}
reader.close();
return data_list.toArray(new String[]{});
}
public static String create_polygon_string(String lon_string, String lat_string) {
/* receives lat/lon of tweet, creates polygon around point, returns poly string in kml format */
double lon_double = Double.parseDouble(lon_string); // convert string from csv to double
double lat_double = Double.parseDouble(lat_string); // convert string from csv to double
double shift_by = 0.0015; // define size of addition/substraction to/from point
String polygon_string;
/* create kml string in polygon style */
polygon_string =
/* opening polygon tags */
"\t<Polygon>\n"
+ "\t\t<extrude>1</extrude>\n" // enabling extruding polygons
+ "\t\t<altitudeMode>relativeToGround</altitudeMode>\n" // define z value as relative to ground
+ "\t\t<outerBoundaryIs>\n" // opening outer boundary tag for polygon
+ "\t\t<LinearRing>\n" // defining boundary as linear ring
+ "\t\t\t<coordinates>\n" // opening coordinate tag
/* perform addition/substraction to point to create polygon around it */
+ "\t\t\t\t" + Double.toString(lon_double+shift_by) + "," + Double.toString(lat_double) + ",100" + "\n"
+ "\t\t\t\t" + Double.toString(lon_double) + "," + Double.toString(lat_double + shift_by) + ",100" + "\n"
+ "\t\t\t\t" + Double.toString(lon_double - shift_by) + "," + Double.toString(lat_double) + ",100" + "\n"
+ "\t\t\t\t" + Double.toString(lon_double) + "," + Double.toString(lat_double - shift_by) + ",100" + "\n"
+ "\t\t\t\t" + Double.toString(lon_double+shift_by) + "," + Double.toString(lat_double) + ",100" + "\n" // repeat 1st coordinate to close polygon
/* closing all polygon tags */
+ "\t\t\t</coordinates>\n"
+ "\t\t</LinearRing>\n"
+ "\t\t</outerBoundaryIs>\n"
+ "\t</Polygon>\n"
+ "\t</Placemark>\n";
return polygon_string; // return finished poly kml string
}
public static String create_profanity_color_string(String tweet, String[] profanity_list) throws IOException {
/* gets tweet string, checks for profanity, returns according kml template style string */
/* get list of profanity from text reader function, passing down file & path of list */
//List<String> profanity_list = Arrays.asList(read_profanity_list(working_dir,profanity_fileName));
/* iterate over profanity list */
for (int i = 0; i < profanity_list.length; i++) {
/* ckeck if tweet contains each word
putting spaces around each word, so that
'passed' is not flagged for containing 'ass' */
if (tweet.contains(" "+ profanity_list[i])) {
/* if tweet contains word, return kml string for according style */
//System.out.println(profanity_list.get(i)); // print out found words
return "<styleUrl>#contains_profanity</styleUrl>";
} // close if statement
} // close for loop
/* if no return from profanity check, return kml template string for no profanity */
return "<styleUrl>#no_profanity</styleUrl>";
} // close method
public static String create_styles() {
/* creating kml style templates for polygons */
String style_section = "<Style id=\"no_profanity\">\n"
+ "<LineStyle>\n"
+ "<width>0.5</width>\n"
+ "</LineStyle>\n"
+ "<PolyStyle>\n"
+ "<color>FF14F000</color>\n"
+ "</PolyStyle>\n"
+ "</Style>\n\n"
+ "<Style id=\"contains_profanity\">\n"
+ "<LineStyle>\n"
+ "<width>0.5</width>\n"
+ "</LineStyle>\n"
+ "<PolyStyle>\n"
+ "<color>FF1400FF</color>\n"
+ "</PolyStyle>\n"
+ "</Style>\n\n";
return style_section;
}
public static void read_convert_save_polygon(String working_dir, String csv_fileName, String profanity_fileName) {
/* opens csv file, writes kml file and cals functions to create polygon and check for profanity,
writes finished kml file */
try {
/* open file csv file via file reader + buffered reader */
File file = new File(working_dir,csv_fileName);
FileReader fr = new FileReader(file);
BufferedReader br = new BufferedReader(fr);
String line = "";
String[] tempArr;
/* Create KML string with document header */
String kml = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
+ "<kml xmlns=\"http://www.opengis.net/kml/2.2\">\n"
+ "<Document>\n\n"
+ create_styles(); // insert style section of predefined styles
String[] profanity_list = read_profanity_list(working_dir,profanity_fileName);
/* iterate over csv file by lines until line is empty */
while((line = br.readLine()) != null) {
/* create Array per line, separated based on semicolon */
tempArr = line.split(";");
/* check if temp array is != id, therefore excluding the csv header line */
if (!tempArr[0].equals("id")) {
/* fill temp string w/ kml placemark syntax + info from iterated array incl. indentation */
String temp_placemark = "<Placemark>\n" // start with opening placemark tag
+ create_profanity_color_string(tempArr[5],profanity_list) // call for profanity check method, returns kml style tag
+ "\t<ExtendedData>\n" // open extended data tag
+ "\t\t<Data name=\"TweetID\"> " + "<value>" + tempArr[0] + "</value> </Data>\n" // sub-tag w/TweetID
+ "\t\t<Data name=\"Tweet\"> " + "<value>" + tempArr[5] + "</value> </Data>\n" // sub-tag w/Tweet
+ "\t\t<Data name=\"Hashtags\"> " + "<value>" + tempArr[3] + "</value> </Data>\n" // sub-tag w/Hashtags, only shows up in GE if hashtags in tweet
+ "\t\t<Data name=\"TimeStamp\" >" + "<value>" + tempArr[6].replace(' ', 'T') + ":00" + "</value> </Data>\n" // sub-tag w/CreatedAt in correct Format
+ "\t\t<Data name=\"UserID\"> " + "<value>" + tempArr[7] + "</value> </Data>\n" // sub-tag w/userID
+ "\t</ExtendedData>\n" // close extended data tag
/* turn timestamp into dateTime format: (YYYY-MM-DDThh:mm:sszzzzzz), same before, then write to correct kml timestamp tags */
+ "\t<TimeStamp id=\"" + tempArr[0] + "\"> <when>" + tempArr[6].replace(' ', 'T') + ":00" + "</when> </TimeStamp>\n"
/* Add polygon string by calling function, returns complete polygon string incl. profanity color stlye */
+ create_polygon_string(tempArr[1],tempArr[2]);
/* append kml string with Placemark info */
kml = kml + temp_placemark;
} // close if statement
} // end while loop
/* close buffered Reader */
br.close();
/* append kml with document/kml closing tags after iteration */
kml = kml + "</Document>\n"+"</kml>";
/* Write KML */
FileWriter fw = new FileWriter("output_polygon.kml");
fw.write(kml);
fw.close();
/* end try block (reading csv), printing error and closing catch block */
} catch(IOException ioe) { ioe.printStackTrace(); }
} // end reader
} // end CSVtoKML class
The created KML files as well as the WMS PNG can be found in the GitHub Repository.
