Comments (10)
Hello @andrewluebke-ms,
We use a third-party library (Apify) to handle the crawling portion of the extension. The discoveryPatterns
input is passed directly into Apify's PseudoUrl
class, so any functionality that will or won't work will be determined by the implementation within Apify.
Here is the documentation to Apify's PseudoUrl
which includes their documentation on "special directives" (regexes) that can be passed in via our discoveryPatterns
input: https://sdk.apify.com/docs/2.3/api/pseudo-url
There are a few rules about how things should be escaped to be handled properly in the Apify library, so it may be worth looking through their documentation to ensure that the regex you're passing in matches what they expect to receive.
This is meant to be positive matching not exclusionary, and in my own testing seems to be fairly quirky about what does and doesn't work. For example I found capture groups to not consistently work the way I expected them to in a standard JavaScript regex, but Apify's docs don't clearly outline why that would be the case.
We hope to be able to implement a solution for excluding URLs in the future that is much more robust and built to do what we're asking it to do, rather than trying to use a mechanism for a purpose it wasn't meant for. We have this on our radar, but currently don't have an ETA for when a better solution will be in place.
from accessibility-insights-service.
This issue has been marked as ready for team triage; we will triage it in our weekly review and update the issue. Thank you for contributing to Accessibility Insights!
from accessibility-insights-service.
This needs more information in order to understand the work required. @lamaks can you help us understand better if there is an exclusion functionality already existing that needs to be wired up or if this would be something new that needs to be built?
from accessibility-insights-service.
Per conversation with Maxim: the functionality doesn't exist yet and it needs implementation, however we support regular expressions that can be used to exclude urls by pattern (parameter discoveryPatterns), which might help. Currently discovery pattern is to for include ULRs, but can be tailored to match to restricted range/pattern, just as any regular expression in general, so it will select subset based on the expression rules
from accessibility-insights-service.
@brocktaylor7 would this approach help for this request?
from accessibility-insights-service.
The team requires additional author feedback; please review their replies and update this issue accordingly. Thank you for contributing to Accessibility Insights!
from accessibility-insights-service.
This issue has been automatically marked as stale because it is marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. Thank you for contributing to Accessibility Insights!
from accessibility-insights-service.
per offline conversation, leaving this as needs investigation since the workaorund of using discovery patterns for exclusion is fairly limited to certain scenarios
from accessibility-insights-service.
This issue requires additional investigation by the Accessibility Insights team. When the issue is ready to be triaged again, we will update the issue with the investigation result and add "status: ready for triage". Thank you for contributing to Accessibility Insights!
from accessibility-insights-service.
Hi! My team is attempting to implement the workaround for exclusionary links in the discovery pattern but are having some troubles. I've detailed an example below that we would expect to work but doesn't seem to work (checked with RegEx checker). Could you provide me with some feedback as to where we might be going wrong and/or some examples of exclusionary RegEx in the discovery pattern?
We are using the Azure DevOps Extension v3.
Here is a throwaway example using google and is similar to what we've tried...
Dynamic Site URL: https://google.com
Discovery pattern: https://google.com/(?!.*FilterMeOut).*
Expected:
https://google.com (link is scanned and crawled)
https://google.com/FilterMeOut (link is skipped)
https://google.com/ValidPath (link is scanned and crawled)
Actual:
https://google.com (link is scanned and crawled)
https://google.com/FilterMeOut (link is skipped)
https://google.com/ValidPath (link is skipped)
(only the first URL is scanned and nothing else is discovered)
The only scenario that fully crawls all our web pages is when we use the provided URL format from the docs:
https://google.com/[.*]. However, we have quite a lot of links we do not want crawled and so excluding them is critical for us.
Thanks!
from accessibility-insights-service.
Related Issues (20)
- [Feature Request] Allow user to control whether URL fragments are ignored when crawling HOT 3
- [Bug] Migrate app from ADAL to MSAL HOT 1
- Getting error logs. Unable to get the index.html file HOT 6
- [Bug] Uncaught Error caused by too many redirects HOT 1
- [Bug] Scanning pages that use Server-Sent Events takes a minute longer than it should HOT 3
- [Bug] Inconsistent summary information in combined report HOT 1
- [Bug] Crawler skips links nested within a shadow tree HOT 1
- False positive #1? HOT 7
- Crawling sometimes fails with NavigationError HOT 5
- WCP Accessibility Scan Crawler Flagging Issues Outside Domain HOT 5
- [Bug] Crawler can detect incorrect HTTP status codes when navigations involving redirect chains are partially cached HOT 4
- [Feature Request] Scan tool should support custom headers HOT 1
- [Bug] scan CLI package reports apify's DEFAULT_USER_AGENT instead of the real user agent in report footers HOT 3
- [Bug] Scan CLI package incorrectly treats 304 responses as errors HOT 1
- [Bug] Can't run as root due to absence of chrome flags HOT 1
- [Bug] Crawling regression between 1.7.0 and 2.1.1 HOT 2
- [Bug] Crawling regression between version 1.7.0 and version 2.1.1 HOT 2
- [Bug] Error: Evaluation failed: TypeError: pQuerySelectorAll is not a function HOT 1
- [Feature Request] Better logging for errors loading pages
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from accessibility-insights-service.