April saw few new features, but those few were quite important. As Microsoft shuttered the Data Market and Bing Translator, we migrated our code to support its successor Microsoft Translate API. Also, we rolled out a new project feature that lets us handle new HTTP methods (such as DELETE or OPTIONS) where needed and previously unavailable. See the full details after the jump!
Microsoft Translate API Integration
Microsoft has long publicized its decision to shut down the Data Market service and with it, its Bing Translator API. As the final interface took shape, we migrated our own interfaces to support the successor Microsoft Translate as well – we are no longer sending requests to the old endpoints, and Bing Translator has been scrubbed from the Account settings as well, instead, you can add your Microsoft Translate API key to enable the new, Azure-based service.
This update also lays the groundwork to exploiting the more advanced features of the Translate API and the Cognitive Services, which will be integrated in the future.
Extended HTTP Method Support
Previously, we have added support for the HTTP DELETE method, which is gaining popularity with new technologies. Unfortunately, Google AppEngine’s infrastructure has imposed one significant limitation on us before (in this case): the inability to add a payload to the DELETE call. This (as well as a number of other limitations) have now been surpassed with a new project attribute that allows us to route requests with unsupported methods through our gateway that allows the required transformation.
This enables us to support DELETE bodies, as well as other, non-standard methods even (should the need arise). However, this attribute is currently only available as a BETA feature, meaning it requires our support to set up.
We also rolled out a plethora of miscellaneous backend tweaks and fixes, aimed at making the crawler more stable, even when dealing with stellar page counts. XML parsing has also been made more stable to deal with potentially-invalid resources and paths, as well as the groundwork for several features to be rolled out the coming month.Dedicated demo for You Tweet