scylla.providers package

Submodules

scylla.providers.a2u_provider module

class scylla.providers.a2u_provider.A2uProvider

基类：scylla.providers.base_provider.BaseProvider

parse(html: requests_html.HTML) → [<class 'scylla.database.ProxyIP'>]

Parse the document in order to get a list of proxies

参数:html -- the HTML object from requests-html 返回:a list of proxy ips

static should_render_js() → bool

Whether needs js rendering By default, it is False.

返回:a boolean value indicating whether or not js rendering is needed 返回类型:bool

urls() → [<class 'str'>]

Return a list of url strings for crawling

返回:[a list of url strings] 返回类型:[str]

scylla.providers.base_provider module

class scylla.providers.base_provider.BaseProvider

基类：object

BaseProvider is the abstract class for the proxy providers

引发:NotImplementedError -- [if urls() or parse() is not implemented]

parse(html: requests_html.HTML) → [<class 'scylla.database.ProxyIP'>]

Parse the document in order to get a list of proxies

参数:html -- the HTML object from requests-html 返回:a list of proxy ips

static should_render_js() → bool

Whether needs js rendering By default, it is False.

返回:a boolean value indicating whether or not js rendering is needed 返回类型:bool

sleep_seconds() → int

Return a sleep time for each request, by default it is 0

返回:sleep time in seconds

urls() → [<class 'str'>]

Return a list of url strings for crawling

返回:[a list of url strings] 返回类型:[str]

scylla.providers.cool_proxy_provider module

class scylla.providers.cool_proxy_provider.CoolProxyProvider

基类：scylla.providers.base_provider.BaseProvider

parse(html: requests_html.HTML) → [<class 'scylla.database.ProxyIP'>]

Parse the document in order to get a list of proxies

参数:html -- the HTML object from requests-html 返回:a list of proxy ips

static should_render_js() → bool

Whether needs js rendering By default, it is False.

返回:a boolean value indicating whether or not js rendering is needed 返回类型:bool

urls() → [<class 'str'>]

Return a list of url strings for crawling

返回:[a list of url strings] 返回类型:[str]

scylla.providers.data5u_provider module

class scylla.providers.data5u_provider.Data5uProvider

基类：scylla.providers.base_provider.BaseProvider

parse(html: requests_html.HTML) → [<class 'scylla.database.ProxyIP'>]

Parse the document in order to get a list of proxies

参数:html -- the HTML object from requests-html 返回:a list of proxy ips

static should_render_js() → bool

Whether needs js rendering By default, it is False.

返回:a boolean value indicating whether or not js rendering is needed 返回类型:bool

urls() → [<class 'str'>]

Return a list of url strings for crawling

返回:[a list of url strings] 返回类型:[str]

scylla.providers.free_proxy_list_provider module

class scylla.providers.free_proxy_list_provider.FreeProxyListProvider

基类：scylla.providers.base_provider.BaseProvider

parse(html: requests_html.HTML) → [<class 'scylla.database.ProxyIP'>]

Parse the document in order to get a list of proxies

参数:html -- the HTML object from requests-html 返回:a list of proxy ips

urls() → [<class 'str'>]

Return a list of url strings for crawling

返回:[a list of url strings] 返回类型:[str]

scylla.providers.http_proxy_provider module

class scylla.providers.http_proxy_provider.HttpProxyProvider

基类：scylla.providers.base_provider.BaseProvider

parse(html: requests_html.HTML) → [<class 'scylla.database.ProxyIP'>]

Parse the document in order to get a list of proxies

参数:html -- the HTML object from requests-html 返回:a list of proxy ips

static should_render_js() → bool

Whether needs js rendering By default, it is False.

返回:a boolean value indicating whether or not js rendering is needed 返回类型:bool

urls() → [<class 'str'>]

Return a list of url strings for crawling

返回:[a list of url strings] 返回类型:[str]

scylla.providers.kuaidaili_provider module

class scylla.providers.kuaidaili_provider.KuaidailiProvider

基类：scylla.providers.base_provider.BaseProvider

parse(html: requests_html.HTML) → [<class 'scylla.database.ProxyIP'>]

Parse the document in order to get a list of proxies

参数:html -- the HTML object from requests-html 返回:a list of proxy ips

static should_render_js() → bool

Whether needs js rendering By default, it is False.

返回:a boolean value indicating whether or not js rendering is needed 返回类型:bool

urls() → [<class 'str'>]

Return a list of url strings for crawling

返回:[a list of url strings] 返回类型:[str]

scylla.providers.spys_me_provider module

class scylla.providers.spys_me_provider.SpyMeProvider

基类：scylla.providers.base_provider.BaseProvider

parse(html: requests_html.HTML) → [<class 'scylla.database.ProxyIP'>]

Parse the document in order to get a list of proxies

参数:html -- the HTML object from requests-html 返回:a list of proxy ips

static should_render_js() → bool

Whether needs js rendering By default, it is False.

返回:a boolean value indicating whether or not js rendering is needed 返回类型:bool

urls() → [<class 'str'>]

Return a list of url strings for crawling

返回:[a list of url strings] 返回类型:[str]

scylla.providers.spys_one_provider module

class scylla.providers.spys_one_provider.SpysOneProvider

基类：scylla.providers.base_provider.BaseProvider

parse(html: requests_html.HTML) → [<class 'scylla.database.ProxyIP'>]

Parse the document in order to get a list of proxies

参数:html -- the HTML object from requests-html 返回:a list of proxy ips

static should_render_js() → bool

Whether needs js rendering By default, it is False.

返回:a boolean value indicating whether or not js rendering is needed 返回类型:bool

urls() → [<class 'str'>]

Return a list of url strings for crawling

返回:[a list of url strings] 返回类型:[str]

scylla.providers.xici_provider module

class scylla.providers.xici_provider.XiciProvider

基类：scylla.providers.base_provider.BaseProvider

parse(html: requests_html.HTML) → [<class 'scylla.database.ProxyIP'>]

Parse the document in order to get a list of proxies

参数:html -- the HTML object from requests-html 返回:a list of proxy ips

static should_render_js() → bool

Whether needs js rendering By default, it is False.

返回:a boolean value indicating whether or not js rendering is needed 返回类型:bool

urls() → [<class 'str'>]

Return a list of url strings for crawling

返回:[a list of url strings] 返回类型:[str]

Module contents