scylla package

Subpackages

• scylla.providers package
  • Submodules
  • scylla.providers.a2u_provider module
  • scylla.providers.base_provider module
  • scylla.providers.cool_proxy_provider module
  • scylla.providers.data5u_provider module
  • scylla.providers.free_proxy_list_provider module
  • scylla.providers.http_proxy_provider module
  • scylla.providers.kuaidaili_provider module
  • scylla.providers.spys_me_provider module
  • scylla.providers.spys_one_provider module
  • scylla.providers.xici_provider module
  • Module contents
• scylla.web package
  • Submodules
  • scylla.web.server module
  • Module contents

Submodules

scylla.cli module

scylla.cli.app_main()

scylla.cli.handle_special_flags(args: dict)

scylla.cli.main(args) → int

scylla.config module

scylla.config.batch_set_config(**kwargs)

scylla.config.get_config(key: str, default: str = None) → Union[str, NoneType]

scylla.config.set_config(key: str, value: str)

scylla.database module

class scylla.database.BaseModel(*args, **kwargs)

Bases: peewee.Model

DoesNotExist

alias of BaseModelDoesNotExist

id = <peewee.AutoField object>

class scylla.database.ProxyIP(*args, **kwargs)

Bases: scylla.database.BaseModel

DoesNotExist

alias of ProxyIPDoesNotExist

city = <peewee.CharField object>

country = <peewee.CharField object>

created_at = <peewee.DateTimeField object>

id = <peewee.AutoField object>

ip = <peewee.CharField object>

is_anonymous = <peewee.BooleanField object>

is_valid = <peewee.BooleanField object>

latency = <peewee.FloatField object>

location = <peewee.CharField object>

organization = <peewee.CharField object>

port = <peewee.IntegerField object>

region = <peewee.CharField object>

stability = <peewee.FloatField object>

updated_at = <peewee.DateTimeField object>

scylla.database.create_connection() → peewee.SqliteDatabase

create a database connection :rtype: SqliteDatabase

scylla.database.create_db_tables()

scylla.jobs module

scylla.jobs.save_ip(p: scylla.database.ProxyIP)

scylla.jobs.validate_proxy_ip(p: scylla.database.ProxyIP)

scylla.loggings module

scylla.scheduler module

class scylla.scheduler.Scheduler

Bases: object

feed_providers()

join()

Wait for worker processes and validator threads

start()

Start the scheduler with processes for worker (fetching candidate proxies from different providers), and validator threads for checking whether the fetched proxies are able to use.

stop()

scylla.scheduler.cron_schedule(scheduler, only_once=False)

Parameters:

• scheduler – the Scheduler instance
• only_once – flag for testing

scylla.scheduler.fetch_ips(q: <bound method BaseContext.Queue of <multiprocessing.context.DefaultContext object at 0x7fb966185828>>, validator_queue: <bound method BaseContext.Queue of <multiprocessing.context.DefaultContext object at 0x7fb966185828>>)

scylla.scheduler.validate_ips(q: <bound method BaseContext.Queue of <multiprocessing.context.DefaultContext object at 0x7fb966185828>>, validator_pool: concurrent.futures.thread.ThreadPoolExecutor)

scylla.tcpping module

class scylla.tcpping.Ping(host: str, port: int, timeout=1)

Bases: object

get_average() → float

get_maximum() → float

get_minimum() → float

get_success_rate()

ping(count=10, sleep=0.3)

class scylla.tcpping.Socket(family, type_, timeout)

Bases: object

close()

connect(host, port=80)

shutdown()

class scylla.tcpping.Timer

Bases: object

cost(funcs, args)

start()

stop()

scylla.tcpping.avg(x)

scylla.tcpping.ping(host: str, port: int, count: int = 10, sleep: float = 0.2) -> (<class 'int'>, <class 'float'>)

Ping a server and port with tcp socket :param host: the hostname :param port: the port number :param count: number of connection tries, by default it is 10 :param sleep: length of sleep time in between sequent pings, by default it is 0.3 :return: a tuple for (average_latency, success_rate)

scylla.validator module

class scylla.validator.Validator(host: str, port: int, using_ssl: bool = False)

Bases: object

anonymous

latency

meta

success_rate

valid

validate()

validate_latency()

validate_proxy()

scylla.validator.get_current_ip()

scylla.worker module

class scylla.worker.Worker

Bases: object

get_html(url: str, render_js: bool = True) → Union[requests_html.HTML, NoneType]

Get html from a specific URL

Parameters:

• url – the URL
• render_js – [whether to render js], defaults to True
• render_js – bool, optional

Returns:

[the HTML string]

Return type:

str

Module contents