insider module in OptScale

[utkarshsri22]

Why insider/insider_scheduler/main.py is hard-coded for SUPPORTED_CLOUD_TYPES = ['azure_cnr'].
It also provides pricing for AWS, so why is this for Azure only?

[nk-hystax]

OptScale uses 2 approaches to get prices from clouds:

1. for not Azure clouds insider-api gets prices by provided filters directly from the cloud while you call some of insider's apis (for AWS

   optscale/insider/insider_api/controllers/flavor.py

   Line 634 in 5694966

   def get_aws_skus(self, executor, flavors, preinstalled, os_type):

   )
2. for Azure cloud pricing data is prepared before insider's api call:

• insider-scheduler once a day creates a task for insider-worker
• insider-worker saves all the prices without filtering from cloud to mongodb (https://github.com/hystax/optscale/blob/integration/insider/insider_worker/processors/azure.py)
• when you call insider's api, the prices in mongo are just filtered by provided filters (see

  optscale/insider/insider_api/controllers/flavor.py

  Line 118 in 5694966

  def get_azure_prices(self, region, instance_family, currency='USD',

  )

[utkarshsri22]

I have a few more questions about this
Q1)Why is it so that Azure Cloud pricing data is prepared before the insider's API call?
Q2)When is the scheduler triggered and why is it only once daily?
Q3)Why are we dumping cloud pricing data in MongoDB? Is it later used somewhere after filtering ?
Q4)Where on UI can we see the use of insider?

[nk-hystax]

Q1) AFAIK this is done because of missing skuId field in expenses for Azure, so we have to combine several filters to find the requested price:

optscale/insider/insider_api/controllers/flavor.py

Lines 124 to 132 in 5694966

	pricing_query = {
	'armSkuName': {'$regex': instance_family},
	'type': 'Consumption',
	'serviceName': 'Virtual Machines',
	'armRegionName': region,
	'currencyCode': currency
	}
	if meter_id:
	pricing_query['meterId'] = meter_id

Q2) Insider-scheduler is a cronjob which is called at 00:00 every day by default (see kubectl get cj insider-scheduler). It's mission is to keep the prices up-to-date. Azure cloud doesn't change the prices very often, so updating prices once a day is enough.
The default schedule is stored here:

optscale/optscale-deploy/optscale/values.yaml

Line 404 in 0bc38e9

schedule: "0 0 * * *"

You may change it by kubectl edit cj insider-scheduler or update your user_template.yml to overwrite the value and update the cluster with this overlay.
Q3) Dumped cloud pricing data is used when you call insider's api to search for needed rows in MongoDB. There are no other places where it is used.
Q4) There are no UI pages that calls insider directly, but it is used by rest_api on "Cost Comparison" page and by bumiworker to calculate savings for recommendations like "Underutilized Instances", "Instances eligible for generation upgrade" etc.

[utkarshsri22]

Where is AWS and azure dumping data in MongoDB ?

[nk-hystax]

Azure: db insider -> collection azure_prices
AWS: db restapi -> collection aws_prices

[utkarshsri22]

Still, I want to understand how AWS pricing is been processed in controllers and handlers and how the data from boto3 is coming and how it is processed such that AWS pricing is coming to us in flavors and other files in insider module ?