Shoveling for Civic Tech Gold

Civic Innovations

Photo courtesy of flickr user SalemSaugusNPS

Unshoveled sidewalks during the winter months are a persistent problem for cities in the snow belt.

For places with significant snowfall, unshoveled sidewalks pose a challenge to public safety and mobility for those that rely on walking (or public transit), and present an especially acute problem for those that have physical impairments. Uncleared snowfall on other kinds of public infrastructure – like fire hydrants – also poses dangers for city residents and public safety workers.

This is a sometimes daunting issue for cities that government officials, community groups and civic technologists perennially struggle with. Over the past several years, a number of civic technology projects have been initiated with the goal of mitigating the problem of uncleared sidewalks and other public infrastructure in cities. Few have been widely adopted, and – if we’re being honest – the impact of these efforts on reducing the problems associated with uncleared snowfall…

View original post 1,389 more words

U-Boats Off the OBX, 1942

This Day in North Carolina History

A tanker sinking off the Outer Banks after being hit by a U-boat.
Image from the National Park Service.

On January 19, 1942, a German U-boat, designated U-123, attacked three ships north of Cape Hatteras. The U-boat was part of what was called the “Second Happy Time,” a campaign during which Axis submarines attacked merchant vessels along the Atlantic coast.

In the early morning hours, the American passenger-freighter City of Atlanta was sunk and survivors were left floating in the icy water; only three crewmen lived. Just before dawn, the U-123 slipped into a group of Allied ships and attacked.

The American tanker Malay was damaged and disabled by gunfire from the U-123’s deck guns. The U-boat then torpedoed a nearby Latvian freighter, killing two of her 32 crewmen and leaving her in to sink.

The U-123 returned to finish off the wounded Malay, hitting her with…

View original post 190 more words

Ultimate guide for scraping JavaScript rendered web pages

IMPYTHONIST

We all scraped web pages.HTML content returned as response has our data and we scrape it for fetching certain results.If web page has JavaScript implementation, original data is obtained after rendering process. When we use normal requests package in that situation then responses those are returned  contains no data in them.Browsers know how to render and display the final result,but how a program can know?. So I came with a power pack solution to scrape any JavaScript rendered website very easily.

Many of us use below libraries to perform scraping.

1)Lxml

2)BeautifulSoup

I don’t mention scrapy or dragline frameworks here since underlying basic scraper is lxml .My favorite one is lxml.why? ,It has the element traversal methods rather than relying on regular expressions methodology like BeautifulSoup.Here I am going to take a very interesting example.I am so amazed after finding that ,my article is appeared in recent PyCoders weekly issue…

View original post 681 more words

Inject Content into a new IFrame

Spare Cycles

In on-line ad delivery we utilize the <iframe> tag quite a lot. There are good reasons for serving ads in iframes. There are restrictions in communication between an iframe and its parent window when the two documents are served from different domains. This helps protect users as well as site publishers from malicious activity.

Iframes also provide advantages in the speed of delivery of an overall page. The parsing and display of an iframe’s content happens asynchronously to the rest of the parent window’s resources. This means that when an iframe is encountered during an initial page load it does not prevent the rest of the page from loading while the content of the iframe is loaded.

At AppNexus we’ve been developing some improved ad delivery mechanisms which leverage the advantages of iframes, in particular we’ve been leveraging the dynamic creation of iframes to serve ad content after some user…

View original post 646 more words

Hide the scrollbar in webkit

Web tricks

Today I decided to improve the looks of Grooveshark, as I run it in a Chrome app window, and I wanted it to sort of melt into the rest of the window decorations. I thought maybe removing the scrollbar from the left sidebar would look good, but how? overflow:hidden hides the scrollbar, but also disables scrolling.

Luckily we have Google, and I found out you can style scrollbars in Chrome with

#element::-webkit-scrollbar { /* css here */ }

So the solution was fairly simple:

#element::-webkit-scrollbar { width: 0 !important }

Scrolling still works, and it looks much better. That said, be careful if using this for others to see. It’s likely that you’ll confuse your visitors when they see no scrollbars!

View original post

Responsible Uses of Open Data

Datapolitan

This week I attended the Responsible Uses of Open Data in Government and the Private Sector conference at NYU. As someone who teaches with open data and lectures on the use of open data in government, I have a keen interest in learning about the latest thinking about the costs and benefits of open data.

The conference seemed tilted towards academics and in particular, academics with a background in law. As an open data practitioner, this made the event more abstract than I’d hoped, with some predictable confusion over the use of particular terms of art:

.@rcalo They're familiar with the concept but not the term. Not everyone here is a lawyer-academic. Please be mindful of that #opendatanyu

— Datapolitan (@Datapolitan) November 20, 2015

The challenge for those of us who work to train local government officials in creating and managing public open data is to translate the concerns and theories of open data into…

View original post 750 more words

Data-Driven Government

Datapolitan

I think like most people, I often use the adjective “data-driven” without having really thought about what it means. I often use it in my data analytics classes as shorthand for how government can be led by data without stopping to discuss the concept or even explain the term.

A recent webcast with Carl Anderson changed that. He was talking about how to create a data-driven organization based on his book. While he drew heavily from his experience in private industry, I began thinking about what this would mean in the context of government and realized I’ve been doing a disservice to my students by not unpacking the term. To be fair, I don’t think anyone has really done this in the context of governing, particularly along the lines that that Anderson advocates in his book. There is an opportunity to develop Anderson’s ideas in the context of government at all…

View original post 843 more words

A cultural landmark is removed, a community reacts

Museums with impact

A few months ago, I wrote this post celebrating the charming grassroots curatorial project thriving on a sidewalk along Duke Street (across from Landmark Mall) in Alexandria, Virginia. I called it the ‘uncurated’ Christmas tree, not because its free-form curation was confined to a Christmas theme (in fact, the decorations changed periodically to reflect the season and were frequently multi-themed), but because it reminded me of my childhood Christmas trees: eclectic, vibrantly disorganized, and organically co-created. The tree was part of my cultural landscape, something that made me feel connected to my neighborhood.

This evening, my husband and I were driving to dinner and were stunned to see that the tree, once growing through the sidewalk and adorned with a random assortment of eye-catching ephemera, had been removed. In its place was a community memorial with notes, candles, even a framed photograph. Comments on Reddit (RIP Duke Street Tree)…

View original post 187 more words

Dive into JavaScript

Python Adventures

I’ve decided to dive into JavaScript. I had some superficial knowledge of it, but nothing serious. I’ve used Flask for my pet projects, but they should also look nice on the client side. So expect some JavaScript and Node.js related posts in the future. But fear not, my favourite programming language is still Python :)

View original post

Download genomes from Genbank

Python Adventures

Problem

For a project, I had to download a bunch of records from the NCBI (National Center for Biotechnology Information) website. A record looks like this: CP002059.1 (almost 5 MB):

LOCUS       CP002059             5354700 bp    DNA ...
DEFINITION  'Nostoc azollae' 0708, complete genome.
ACCESSION   CP002059 ACIR01000000 ACIR01000001-ACIR01000216
VERSION     CP002059.1  GI:298231532
DBLINK      Project: 30807
...
ORIGIN
//

I needed this data in text format.

Solution #1
My first idea was to download the page with wget. However, I was surprised to see that the downloaded file was less than 100 KB instead of 5 MB! When I looked at the source, it turned out that it’s full of AJAX calls. That is, the browser downloads this short HTML and then it is expanded. If you save the page with File -> Save as…, you have the complete HTML but how to automate the download process? How to get the post-AJAX…

View original post 104 more words