[FEATURE] Tags/users search about instamancer HOT 6 CLOSED

Thanks for the suggestion.

You're right that firing DOM events is something that I'm mostly trying to avoid for this project. But just as a point of clarity regarding the project's ideology, interacting with the DOM in a limited way is fine, but what I want to avoid is reading data from page elements. Data should always be obtained directly from the API either by intercepting responses, or reading the API responses from the page's memory.

For this feature, there are three ways I can see to implement it:

Method 1
Use the browser, but avoid querying the DOM by using keyboard events:

import {launch} from "puppeteer";

(async () => {
    const browser = await launch({headless: false});
    const page = await browser.newPage();
    await page.goto("https://instagram.com/explore/tags/puppies")

    await page.mouse.click(1, 1);
    await page.keyboard.press("Tab");
    await page.keyboard.press("Tab");
    await page.keyboard.type("test");
})()

This method avoids querying the DOM which could be restructured at any time. However it's reliant on the input box always being the second element that can be tabbed to, and that clicking 1x,1y won't fire an unwanted event.

Firefox has a configuration option that allows you to tab directly into the first input box, but Chrome doesn't have such an option.

Method 2
Query the DOM:

import {launch} from "puppeteer";

(async () => {
    const browser = await launch({headless: false});
    const page = await browser.newPage();
    await page.goto("https://instagram.com/explore/tags/puppies")
    
    await page.click("input[type='text']")
    await page.keyboard.type("test");
})()

Although usually I'd want to avoid this, in this case it works quite well. An explore page is unlikely to contain any input text elements other than a search box, so using input[type='text'] should be quite resistant to changes.

Method 3
Query the API directly:

import {launch} from "puppeteer";

(async () => {
    const browser = await launch({headless: false});
    const page = await browser.newPage();
    await page.goto("https://www.instagram.com/web/search/topsearch/?context=blended&query=test")
})()

At this moment, there doesn't seem to be any sort of authentication mechanism or rate limiting for this API endpoint, and so we can just query it directly. If you like, you can skip puppeteer / instamancer altogether and just use your own http client to query this in your application.

So if you'd like to submit a pull request for this feature, I think the method that best fits this project is number 2. It is the least likely to fail in the event of changes to the frontend (restructuring), or backend (adding rate-limiting, authentication).

If you do implement it, make sure you're obtaining data by intercepting the response from the API, not reading the DOM elements:

from instamancer.

Looks like instagram has removed the ability to search without signing in.

from instamancer.

Maybe not? They might've reversed the change but it appears to be working now? I'm not sure what cookies / session params are causing this:

A blank profile like instamancer uses seems to work correctly

from instamancer.

Strange, can not reproduce this too.

If they will close the search for a not authorized account, do you mind if I will do an authorization feature in instamancer? Or it is against the philosophy of the project?

On the other hand, we can bring back location feature with authorization.

from instamancer.

For now I can't include auth in the project due to the context in which instamancer is being used, although that may change in time.

You should be able to use the new plugin system to develop your own way to log in. It works pretty well in scripts but you need to actually clone+edit+build to use plugins in the CLI for now.

from instamancer.

Actually, that wouldn't work just yet because the event emitter isn't capable of waiting for you to log in before scraping.

from instamancer.