Skip to content

Multiple Perf Improvements #80

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tgrigsby-sc wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Multiple Perf Improvements #80

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
06d7996
update instance type to standard m5. 4GB mem/cpu
tgrigsby-sc May 20, 2021
a559ec1
change to on demand instance type
tgrigsby-sc May 20, 2021
923bacb
bug fixes for low zoom and raw tile docker builds
tgrigsby-sc May 20, 2021
714b0ea
add tile specifier and sort by raw tile size
tgrigsby-sc May 20, 2021
778c09a
change zoom_max to 6 to allow better grouping of fast high zoom jobs
tgrigsby-sc May 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 2 additions & 31 deletions batch-setup/batch_setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,34 +148,6 @@ def batch_setup(region_name, run_id, vpc_id, securityGroupIds, computeEnvironmen
boto_iam, boto_ec2, instanceRoleName, instanceRoleName)
print("Using ECS instance profile %s" % instanceProfileArn)

# Create the spot fleet role
# https://docs.aws.amazon.com/batch/latest/userguide/spot_fleet_IAM_role.html
spotIamFleetRoleName = "AmazonEC2SpotFleetRole"
Copy link
Copy Markdown
Member

@iandees iandees May 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be neat if we could think of a way to keep support for spot instance types for folks where Spot is cheaper than on-demand. On the other hand, it might be enough to point to this commit and say "undo this one".

Copy link
Copy Markdown
Contributor Author

@tgrigsby-sc tgrigsby-sc May 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can certainly do it, it's just some plumbing

spotIamRoleDocument = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "spotfleet.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
spotIamFleetRoleArn = get_or_create_role(
boto_iam, spotIamFleetRoleName, spotIamRoleDocument)
print("Using EC2 Spot Fleet role %s" % spotIamFleetRoleArn)

spotFleetTaggingRolePolicy = find_policy(
boto_iam, 'AmazonEC2SpotFleetTaggingRole')
boto_iam.attach_role_policy(
PolicyArn=spotFleetTaggingRolePolicy['Arn'],
RoleName=spotIamFleetRoleName,
)
print("Attached Spot Fleet Tagging Role to EC2 Spot Fleet role %s"
% spotIamFleetRoleArn)

# Create a compute environment for raw tile rendering
try:
response = boto_batch.describe_compute_environments(
Expand All @@ -192,11 +164,11 @@ def batch_setup(region_name, run_id, vpc_id, securityGroupIds, computeEnvironmen
state="ENABLED",
serviceRole=serviceRoleArn,
computeResources=dict(
type='SPOT',
type='EC2',
minvCpus=0,
maxvCpus=max_vcpus,
desiredvCpus=0,
instanceTypes=["r5"],
instanceTypes=["m5"],
# although this is called "instanceRole", it really wants an instance _profile_ ARN.
instanceRole=instanceProfileArn,
tags={
Expand All @@ -205,7 +177,6 @@ def batch_setup(region_name, run_id, vpc_id, securityGroupIds, computeEnvironmen
'cost_resource_group': run_id,
},
bidPercentage=60,
Copy link
Copy Markdown
Member

@iandees iandees May 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can probably be removed, too.

spotIamFleetRole=spotIamFleetRoleArn,
subnets=subnet_ids,
securityGroupIds=securityGroupIds,
)
Expand Down
Loading