Test warc file download

Latest commit. Git stats commits. Failed to load latest commit information. Enable writing block digests for warcinfo records Aug 11, New record HTTP http semantics Oct 5, Windows Fixes Jul 12, Mar 4, Mar 6, Oct 27, Mar 29, View code.

Install with: pip install warcio This library is a spin-off of the WARC reading and writing component of the pywb high-fidelity replay library, a key component of Webrecorder The library is designed for fast, low-level access to web archival content, oriented around a stream of WARC records rather than files. For example, the following prints the the url for each WARC response record: from warcio.

Code of conduct. Releases No releases published. Sponsor this project. Publication date While testing Version 2 of the Archive Team Warrior virual download appliance, a selection of Tumblr blogs were downloaded over the course of a day.

This gb collection of mostly random sites are being stored here in the off-chance later generations want to check them out, or if something obscure was caught before being properly archived later. Addeddate Firstfiledate Identifier archiveteam-tumblr-test-warc Lastfiledate Pages Scandate Year There are no reviews yet.

Be the first one to write a review. Archive Team. Web Crawls. What do you want to parse it into? Andrzej Dolyle I want to access to its content to index the contents of its web pages. That doesn't really answer my question, in the way I intended. To parse something, means to convert it from a text representation, into a suitable object model. Without knowing that, I don't know what you're concretely trying to do here. AndrzejDoyle it is not text it is an html body with warc header I want to have only content of html such as title and content of the page.

As I understand, you have it in a file. That means it's a sequence of characters - text. You need to parse it if you want to turn it into some Java object. I'm asking you specifically what Java class you want to represent this data?

Show 3 more comments. Active Oldest Votes. Improve this answer. Add a comment. Derek Chia Derek Chia 5 5 silver badges 12 12 bronze badges. Vanaja Jayaraman Vanaja Jayaraman 3 3 silver badges 16 16 bronze badges.