News

Changes and announcements

September 2022 (v1.87)

Hi everyone! Welcome to the September 2022 release of SEO Studio. This update brings a large number of changes and improvements which add more value to your tools and are sure to improve both you and your users' experiences. Here are the most notable changes:

We also have several smaller changes in this release that deserve some attention:

Finally, we have a list of other improvements, tool changes, and bug fixes that you can check out here.

Unicode domain names now supported

You can now enter unicode domain names (such as ๅใŒใƒ‰ใƒกใ‚คใƒณ.com) into the tools. The user interface has been updated to properly display them, including in saved sites and across the admin panel. The parsing logic for domains and URL inputs has also been completely reworked to improve reliability.

For those who aren't familiar with how these domain names work, they are basically converted into ASCII. For example, the domain above will translate to xn--v8jxj3d1dzdz08w.com. Even if you enter a domain in this xn-- format, the user interface will convert it to unicode. This also applies retroactively to existing domains in your system.

Unicode domains

Download tool results

Buttons have been added to the top right of tool result tables for copying and downloading as CSV. The resulting clipboard content and files should be compatible with all spreadsheet applications. New color customization options have also been added to help you change the appearance of these buttons.

There are some important notes about the copy button โ€“ it won't work for non-HTTPS websites and will be hidden instead. Additionally, the copy button will not appear for older embedded tools โ€“ you will need to generate new embed codes to begin using it, or add the allow="clipboard-write" attribute to the old iframes manually.

I've never implemented a feature quite like this before, so if you run into any issues or have any suggestions on how this feature can be improved, please do let me know.

Copy & download

This release adds new country dropdowns to the Google SERP, Bing SERP, Top Search Queries, and Competition tools. These searches will be emulated in the target country for realistic results. It's also worth noting that these dropdowns all contain a blank option at the very top which will revert to the old "global-ish" behavior.

Country dropdowns

Local currencies in keyword research

The keyword research tool now displays the cost-per-click value in both USD as well as the target country's local currency. This only applies when the local currency is different than USD, otherwise the new column is not shown.

This new change requires that currency exchange values are updated periodically. To accomplish this, I've made such data available through my API for free. Your script will automatically fetch the latest currency rates every few days to always display accurate results.

In the future, I will add optional, alternative data sources for currency exchange to avoid overreliance on my API.

Local CPC

Sitemap and robots.txt management

You can now manage the sitemap.xml and robots.txt files for your website from Content » Files. If these files do not exist, you will be presented with an option to generate them with default content. I've also leveraged the built-in CodeMirror editor to add syntax highlighting.

In addition, a new API has been added to the script for dynamically working with the sitemap. The sitemap will be updated automatically when changing permalinks, tools, extensions, or when moving the website to a new URL. When this happens, you will see a notification in the admin panel asking you to review the changes.

It's safe to modify the sitemap yourself using the new editor. Your changes will be fully validated before saving, and the script won't overwrite anything that you add, remove, or edit yourself. And if you ever want a fresh start, simply clear out the file and you'll be offered to generate a new one again.

Sitemap editor

Rate limiting & captchas

The script now has a full rate limiting system. When enabled, requests are logged and rate limited for the login, registration, contact, and tool pages. You can also rate limit search network and keyword research requests on top of this, to help preserve any associated quotas.

There's a full dashboard for managing this at Settings > Rate limiting. You can set the individual rate limits for each type of request across a range of different intervals. If the user exceeds the rate limit for any of those intervals, they will be blocked from that type of request for the specified duration (in minutes).

You can now also add captchas to the script from that same dashboard. When captchas are enabled, users can complete a captcha to immediately circumvent the block. Otherwise, they will need to wait it out.

โš™๏ธ Developer notes: This feature uses the database to record the IP address for each request. It does not use cookies. More options for this will be added in the future. Make sure your cron job is running to clean this up!

Captcha

Keywords Everywhere integration

Keywords Everywhere is a premium data source for keyword research which derives its data from Google's Keyword Planner. Instead of subscriptions, they offer rechargeable credits starting at $10 per 100,000 keywords.

If you're willing to pay for those credits, you can now enter your API key into SEO Studio and it will automatically use their data to unlock competition and trend data for the following countries: Australia, Canada, India, New Zealand, South Africa, United Kingdom, and United States.

The script will automatically track your available credits, and can even send you an email notification when your balance drops below a set threshold. Plus, if you ever run out of credits, the script will automatically revert back to using the free, built-in data source instead.

You can manage this integration from the new Services » Data sources dashboard. This is completely optional, but highly recommended for any commercial or high-traffic websites.

Keywords Everywhere

Search proxying changes

I've made some notable changes to the free built-in proxying service for search queries with this release.

  • The service is being rebranded from "Google Network" to "Search Network".

  • Requests for keyword suggestions (e.g. from the keyword research tool) are now counted against your daily quota. This shouldn't affect most users, as these requests are blocked far less frequently than search queries.

  • In addition to your daily quota, there is a new limit on the number of requests you can send to the service per minute. This limit can be calculated by multiplying your daily quota by 0.05. I'll increase this limitation after I'm able to complete some much-needed network upgrades.

  • Private or local installations are now eligible to enable the service, albeit with a smaller quota. This should help with testing as well as private usage.

  • The service has been upgraded to support localized searches for both Google & Bing.

The search network has seen explosive growth in both supply and demand recently, and it's struggling to keep up. This may be apparent at times as it will cause search-based tools to perform a bit slower than usual. Please bear with me as I redevelop the network with a more performant software stack to handle the increased load.

Note that hitting your quota or rate limit doesn't mean your tools will fail. The script will fall back to performing the search locally if it can't complete the request using the search network, so tools will still work normally.

Stay tuned for some more exciting news in the coming months. As always, you can check your current proxying status and daily quota from the services section of the admin panel.

Google native parsing

This release adds a brand new parser for Google search results that uses libxml on supported servers. The result is a drastic decrease in parsing time (from 0.8 seconds to 0.05 seconds in my tests) for a full page of 100 results. The tools are still a bit slow due to the sheer volume of data moving around, but I'll continue working on it.

There is also a new experimental parsing strategy that uses accessibility elements to almost immediately identify core features of the search results page, as opposed to the old strategy which went in completely blind. This means search result parsing should be far more reliable now, although we may need to squash some bugs here and there first.

I've tested a wide range of queries and search result pages without any issues, but if you encounter anything strange or unexpected in search-based tools after this update, please report it to me.

Automatic language detection

The script now recognizes the Accept-Language header and uses it to automatically select the correct language according to each user's preferences. Modern browsers allow users to specify multiple accepted languages, and to assign a priority to each.

The script correctly handles these prioritized lists and will automatically pick out the highest priority and closest matching language that exists in your system based on locales.

Note that locales don't need to be an exact match. For example, if you have a language set to es-mx but a user accepts es-es instead, they will still be served the es-mx language if an exact match can't be found.

New tool embedding settings

I recently added a settings page for tool embeds which could be used to disable embedding and to whitelist domains. This release builds upon these settings to give you even more control.

  • Added strict mode โ€“ prevents embeds from loading if the parent website cannot be determined
  • Added new options to easily choose where tools can be embedded
  • Added new fields to set custom HTML for headers and footers in embeds

Tool cache now supports unicode keys

Tool result caching has been improved to support unicode keys. This fixes some issues where certain inputs could cause a collision with cache from other inputs in the same tool, and will improve overall reliability.

Redirect after login

The authentication flow has been improved to redirect users back to their original page after successfully signing in. This improves some confusing user interactions and allows you to share links to protected routes, such as tools which require login to access.

New color customization options

I've added three new color customization options to the admin panel, allowing you to easily edit the body background color, as well as the background and border colors for panels. Panels are the white blocks found around all tool inputs and results. In addition, changing the panel borders will affect certain table borders for a more consistent look.

Check out the new color options in the customize section.

Color customization

New security settings page

You can now manage your script's security settings from the new Settings » Security page. These settings help address the multiple attack vectors associated with scraping from user-provided links.

  • Disallow local network access โ€“ Prevents users from running tools on private IP addresses, and from entering hostnames that resolve to private IP addresses.

  • Resource download size limit โ€“ Limits the number of bytes that can be downloaded from a single resource, such as from the link a user enters into a tool.

These policies only apply to HTTP requests sent with the new UserWebRequest class, which resolves and caches DNS on its own and tracks the response size (both body and headers) for enforcement. Expect further updates to continue on this work soon.

Require login for specific tools

You can now require users to be logged in before using certain tools. You can find the toggle for this in the settings tab for each tool under Content » Tools. This option will also prevent affected tools from being embedded.

When a tool requires login, and the current user is not logged in, the tools page will show the tool's icon in grayscale to signal that it is not accessible. Clicking on the tool will redirect to the login page, and after the user successfully signs in, they will be redirected back automatically.

Other changes

Tools

  • Backlinks: Increased the number of results shown to 350 (from 250)
  • Top Referrers: Increased the number of results shown to 200 (from 100)
  • Top Search Queries: Added IE, IT, MX, NL, UK, SG, and ES to the list of countries
  • Bing SERP: Increased the number pages crawled if the target website is not found
  • Headers: Redacted set-cookie headers from output to prevent abuse
  • Sitemaps: Improved reliability and crawling performance

Improvements

  • Improved the layout for tool result tables on small screens
  • Improved the visibility of cards, forms, and tables throughout the admin panel
  • Improved the design of success alerts in the admin panel
  • Improved performance and memory efficiency of the built-in WebRequest library
  • Added a new tool settings page for tool-related configuration
  • Added a button to change the icon theme for all tools at once
  • Added a configuration option to disable favicons in tool results
  • Added noindex and nofollow to tool result pages and embeds
  • Added syntax highlighting to the Custom HTML settings page using CodeMirror
  • Added an expiration timestamp to URLs in cookie-free site input mode to prevent abuse
  • Added a check against <img> ad banners showing multiple times on the same page
  • Added a label to the tool usage admin page to see when and where tools are embedded
  • Error reporting keys have been added to metadata and can now be rotated with each update
  • Error reports will no longer be sent if an update is known to be available
  • Error reports will no longer be sent if the originating file was modified since the last update

Bug fixes

  • Fixed indexed page counts sometimes being reported as 0 for Google Search
  • Fixed the count parameter being ignored for Bing Search
  • Fixed some rare errors in the Search Network that went unreported
  • Fixed some undefined property warnings in the API client
  • Fixed some missing query parameter checks to the admin panel
  • Fixed an issue with the sitemap tool that caused fatal errors when certain links failed to load
  • Fixed an unhandled error when a language is missing from the database
  • Fixed issues with punycode regular expressions on some servers
  • Fixed poor spacing around site inputs on mobile devices
  • Fixed missing right margin on tool form labels
  • Fixed custom favicons not showing up in the admin panel
  • Fixed an incorrect colspan when there are no results in the competition tool
  • Fixed the "website picker foreground" custom color only working on the placeholder
  • Fixed an error on the search service in the admin panel
  • Fixed broken path to the background URL update script in the admin panel
  • Fixed permalinks remaining "on" even if there was an error during activation
  • Fixed a plugin not found error in the contact page quick start utility for fresh installations
  • Fixed a regex bug that caused the Custom HTML settings page to hard post
  • Fixed disabled tools still being accessible in embedded mode
  • Fixed missing margins around admin panel pagination buttons
  • Fixed missing margins around icon in tool errors
  • Fixed incorrect locale format in the public website header
  • Fixed broken icon set conversion for the new page authority tools
  • Fixed an issue where conflicting permalinks wouldn't be correctly prioritized
  • Fixed issues with tooltips not always showing on sparklines