Find original and updated publication dates of any web page. From the command-line or within Python, all the steps needed from web page download to HTML parsing, scraping, and text analysis are included.
In a nutshell, with Python:
>>> from htmldate import find_date >>> find_date('http://blog.python.org/2016/12/python-360-is-now-available.html') '2016-12-23' >>> find_date('https://netzpolitik.org/2016/die-cider-connection-abmahnungen-gegen-nutzer-von-creative-commons-bildern/', original_date=True) '2016-06-23'
On the command-line:
$ htmldate -u http://blog.python.org/2016/12/python-360-is-now-available.html '2016-12-23'
htmldate finds original and updated publication dates of web pages using heuristics on HTML code and linguistic patterns. URLs, HTML files, or HTML trees are given as input. It provides following ways to date a HTML document:
- Markup in header: common patterns are used to identify relevant elements (e.g. link and meta elements) including Open Graph protocol attributes and a large number of CMS idiosyncrasies
- HTML code: The whole document is then searched for structural markers: abbr and time elements as well as a series of attributes (e.g. postmetadata)
- Bare HTML content: A series of heuristics is run on text and markup:
- in fast mode the HTML page is cleaned and precise patterns are targeted
- in extensive mode all potential dates are collected and a disambiguation algorithm determines the best one
The output is thouroughly verified in terms of plausibility and adequateness and the library outputs a date string, corresponding to either the last update or the original publishing statement (the default), in the desired format (defaults to ISO 8601 YMD format).
- Should be compatible with all recent versions of Python (currently 3.4 to 3.9)
- Designed to be computationally efficient and used in production on millions of documents
- Batch processing of a list of URLs
- Switch between original and updated date
Markup-based extraction is multilingual by nature, text-based refinements for better coverage currently support German, English and Turkish.