Specialties: Development, developer workflow, devops, linux, Debian, CI, benchmarking, profiling, bug fixing, performance, scalability, ops planning, capacity. Any client can connect to any server because they negotiate to determine They have a mirror video driver built into their system that helps speed up. Google supports virtual display devices on Linux instances and on Windows instances that use any xbased Windows images v or later. ULTRAVNC O QUE Вы можете прийти к нам.
It spikes, though. So I should probably actually do it. But B, it runs on Heroku. So it's self. It's kind of weird, Heroku logs going to Heroku, then going to you guys. So, yeah, super cool. So once the data's in BigQuery, what are you actually doing with it there? So we have a bunch of standard queries that we run to find some issues. So we either run them directly in your dashboard, or we use something called Redash, which, I think, you actually recommended to me a while ago.
But Redash, for those of you that don't know, is an open source dash boarding system. It's super cool for that. So we use that. We have a bunch of queries that run regularly to find certain things that we're looking for, do stats. And it's super awesome because it's cheap to run on you guys and it's also fast. Do you actually have a bunch of different tables? Is it just one humongous table with millions of rows? This is the new mistake, which I need to fix, which I was reading the docs about recently.
When I first started, I didn't realize that we should do named, date version tables. It turns out-- at least in the past-- that was impossible to fix without just making a new set of tables, which I didn't really want to do. So I think I read the docs correctly, saying that I can actually do that now without taking offline, which looks kind of cool. So, anyway, I haven't gotten around to it, because the cost is actually low enough that I don't care yet.
Is that correct? There is some discount if the data doesn't get added to after a period of time. Anyway, most of our interest I'm actually not sure. But Google it yourself. MARK: One cent per gigabyte per month. And it's also awesome. We learned BigQuery from him. MARK: That's why we did the podcast. And it's working, which is kind of cool. RUSSELL: So, normally we get-- we don't really run queries over the full data set unless there's some security thing that we're trying to investigate.
So almost all our queries-- I can't remember what it's called. But anyway, almost all our queries are done by that. So depending on what you're doing, they're normally either a day or a week's worth of stuff But only on rare circumstances do we query all of the data ever.
So it's normally if we have something that somebody's, like, oh, this thing happened two months ago. Can you help us? So I'm, like, yeah, maybe we can. It turns out. Yeah, we don't normally query the whole data set. So things are pretty quick, normally under a minute.
We queried earlier to find out how many words were in the table today. And that was the longest query I've ever seen, which was just over a minute. And that's because it counted every single thing ever, which is kind of still really cool. MARK: So you were talking about your visualizations and you were saying you were using it to find certain issues and things. Can you talk a bit more about what sort of issues that you're looking for, or how are you using that in the MARK: Yeah. So we're also using another awesome product to do this, which seems to actually be the better way of doing it.
Short version, we're looking at trends in memory usage or loads. Can we spot any trends that are outliers and then dig into why this happened. So we're actually using a service called honeycomb. But I can super recommend those guys. Short version, we're trying to find-- this bunch of requests was slow. Why was that? And then, for instance, we were investigating the other day and found one of our Dyynos-- which I guess Roku parlance for a machine running a process-- was using way more swap than all the other machines for no apparent reason.
So digging into that, even finding that, was only possible using BigQuery and Honeycomb. MARK: So most of your data really is about machine analytics and seeing what your actual things are doing? RUSSELL: Yeah, we haven't moved to-- and there's another thing that we're working on, which is to move some of our data or a lot of our most important tables from Postgres into BigQuery.
And the main reason we want to do that is for doing analytics. So turns out we have too much data to be able to query the entire thing at a reasonable speed on Postgres. So we've got coming up to or gigs of data, which actually doesn't seem like that much. But turns out, is enough. It's expensive to run at a fast speed when we're doing stupid queries.
So anyway, short version, we're working on a way of exporting that data direct into BigQuery, which will mean that we can do the more insane queries faster, which is going to be cool. Do you have any other piece of advice for people that are starting to migrate, that they have a huge database relational. And they're like, should I move to BigQuery?
Yeah, basically the biggest limitation when I started doing it was-- well, I guess I did read the docs. But I just didn't bother doing a bunch of stuff because I wanted to get started. But the biggest thing I would say now is make sure you read the docs properly and probably go with the standard SQL stuff, which seems to be the new version.
I'm still on the older version. The standard SQL seems super awesome because it removes one of the major limitations of BigQuery, which is updating stuff. Now you can treat it almost like a proper database. I don't mean that meanly, by the way. It's not just depend only now. You can actually update things. It totally makes sense if you're talking about something to analyze logs.
RUSSELL: Perfect use case for us, but horrible use case for the analytic stuff for us, because we need to just drop it every time we want to update it again. Which, I guess, is also possible. I would say definitely look at the streaming API. It's pretty awesome.
It also means stuff's really fast. There's a really good go library for it that I found, which I will get put in the podcast notes. And then I guess the only other thing-- what do you want to do to wrap up? The only other thing I was going say was I'm in Reinvent. I'm speaking at Reinvent about MTurk. So if anyone wants to come say hi and do that. So if anyone wants to say hi there, do it.
Well, thank you so much for taking the time today to talk with us. That's a very interesting interview. I will have a bunch of links to all the different projects that you have mentioned. And hopefully we'll also have links to the open source versions of your projects.
MARK: Awesome. Well, thank you. Thanks, guys. MARK: Thanks. It was a pleasure and a joy to have Russell Smith on with us today. I definitely appreciate hearing about people building interesting things. Especially, I think, across cloud, I think that's really interesting to see how people are doing that sort of stuff. And it was really good. And the amount of data he's processing, it's decent. It's a decent amount. Yeah, when he was walking away we had a quick chat. And one of the things that I was saying is, oh so, you have 25 billion rows.
Are they very short rows? It's like, oh no. They have ten or more fields. Oh wow. And some of the fields have JSON inside. So there's a decent amount of data. Yeah, very cool. So yeah, looking forward to all the open sourcing of those tools. MARK: Yes. Very good. They make you very happy. All right cool. So let's get into our question-- apparently questions-- of the week.
Yeah, you raised the first one, which I think is an interesting one and got me very excited. Which is, how to react to email from Google Cloud. So say I-- I don't know. This is always a fun one. I'm going to get sent an email by some third party, maybe you for example. And I want to be able to automatically do something about it once I receive it. How can I find out when that email comes to me, so I can pause it's data and do stuff like that?
And it was pretty funny because they need to do something very complex on the compute engine. And that very complex job needs to be done every single time that someone could update a database, that it was publicly available, built by NASA, actually. The impressive thing is that the way they going to notify people there's new data in the database is by sending emails.
MARK: Of course. So you have someone that was receiving that email that then could go and run a script that could start everything. And I was like, I'm pretty sure there's a better way to do this. And there's many ways of doing it. One of them could be, run your own web server or run your own mail server. MARK: Let's don't do that. That doesn't sound like fun. Then another thing you could do is basically use IMAP to connect from your script that runs regularly.
Like, for instance, using Chrome. And every hour it checks if there's new email. And then the email starts doing stuff. But then the problem is that, how often do you check? MARK: You're basically polling. So it's going to not be perfect. There is actually a very, very easy solution, which is just using app engine. MARK: So app engine can receive email? So, you actually get the same way-- you get a URL to host your web server, which is like your appid.
You also get an email, which is whatever you want at yourappid. So if you send an email to that address, what will happen is that the email will be sent as a post, HTTP request, to your application. So now if your instance is not running-- like your application is not running because you don't have any traffic, which is probably normal if you're only receiving mail once a day.
What will happen is that when you receive that request, your new instance will start. And you will treat that and then you do whatever you want. And one of the things that you can, as we said at the beginning of the podcast, was that you can start a new computer engine instance. And then from there you do whatever you need. MARK: I'm just actually genuinely blown away that app engine can send me email. I didn't realize it could do this. And it's so cool. It is really not new. It is very simple.
And if you know it's there it is good. MARK: My mind is literally blown. It is Literally blown. And talking about app engine, we also have a second question of the week coming from our friend, Stu Swartz, or Schwartz. Tell us how you pronounce it. So, it is about app engine services, which for people who have been using app engine for awhile, you might have known as app engine models.
And he's asking when should I use app engine vs. Is this a scale issue, an approach issue, implementation issue? And then the second question, which is more related to app engine services, is should I have a bunch of services? Should I have a bunch of projects and one service per project? The answer to the second part of the question is, well, have as many services as you need.
Micro services are awesome. MARK: And when you say services, we're talking about modules. So you can have, basically, a different app-- it's basically a different app engine application. But they share the same resources. So if you want to make them communicate through You can.
But also you can use task queues and all of these things. And it's very easy to communicate because they're all in the same project. When to use more than one project. Normally, I could reserve that to a different environment. So you have like prod, QA, stuff like that. Then those are good because that way you're not sharing the data store, which is probably a good idea. So it is a good place to separate. But you should have all your production services in one single project.
It's much easier to handle. And then about the question, when to use app engine versus GKE, maybe you can give us a little bit of background on that. MARK: Yeah, so this is an interesting question. I'll hearken back for a more detailed response to our episode number two, Compute as a Continuum, where we go into much nuance detail about it.
But I feel like it's really more a question of how much flexibility and control do you want. App engine, standard and app engine flexible will give you slightly different versions of that. Whereas app engine standard platform as a service, we dictate a lot of things about how you do stuff. But that gives you great scale. Which is really powerful. Flex, we give you a few more configuration options, a few more bits and pieces so you can run pretty much anything you want on it.
But now you can't scale to zero. We don't spin up quite as fast. So there's the trade off. MARK: But if you want, still, the sort of capabilities such that things stay up. If they fall over, they come back. You still want to be able to deploy with a rolling update, things like that, GKE is a great place for that. But you now have even more tweaks and knobs that you can pull. So you'll have a cluster of machines, in that case. You pay for that cluster of machines.
You can order or scale it to a degree. But then you can do things like resource management or have jobs or scheduled jobs, as well, once that comes out of alpha. But you have way more control. And you can say, maybe I want to have stuff with a whole bunch of memory in it, which is way more than I'd be able to get on app engine or something like that. Persistent volume amounts, if there's state that I want to be able to put in places, things that you can't necessarily do as easy on app engine.
So it's probably really down to what sort of application you're building, and then what sort of capabilities you really want to be able to have at hand. That way I can forget about all the DevOps. And I just deploy it, and I know it will be running there forever. And that makes me happy. But if it's something that I'm going to need to be like, oh no, I have all of these services and there's dependencies across them. And I want them to be able to scale them independently and update them independently, and all that stuff, then GKE is probably the best place.
MARK: Seems pretty reasonable. MARK: I've got a lot of trouble. I say that with some sadness. But I'm doing some fun events, so that's pretty cool. So I will be at Siege in Atlanta on the 8th of October. That's going to be a fun gaming conference. After that, I'll be at Connect Tech on the 20th. I'll be back in Atlanta, again, on the 20th. That's a general web dev conference.
And then I'll be at Game a Con, another game conference on the 27th. So I have a fair bit of travel in October. MARK: And yourself? What are you doing? And I'm going to be running a workshop. And you're all invited. Yeah, it's about how to be a highly scalable web applications with go on a pension. Everyone is welcome. So we'll have a link on the show notes to that. And then I'm also very excited about something else, which is not traveling but kind of.
I'm going to be doing a remote, a remote meet up. That will be on Monday after that, so on October 10th. And it's going to be kind of the same. So it's a mini workshop. It's how to build a web application with Francesc.
Sentiment analysis and classification of unstructured text. Custom machine learning model training and development. Video classification and recognition using machine learning. Options for every business to train deep learning and machine learning models cost-effectively.
Conversation applications and systems development suite for virtual agents. Service for training ML models with structured data. API Management. Manage the full life cycle of APIs anywhere with visibility and control. API-first integration to connect existing data and applications. Solution to bridge existing care systems and apps on Google Cloud.
No-code development platform to build and extend applications. Develop, deploy, secure, and manage APIs with a fully managed gateway. Serverless application platform for apps and back ends. Server and virtual machine migration to Compute Engine. Compute instances for batch jobs and fault-tolerant workloads.
Reinforced virtual machines on Google Cloud. Dedicated hardware for compliance, licensing, and management. Infrastructure to run specialized workloads on Google Cloud. Usage recommendations for Google Cloud products and services. Fully managed, native VMware Cloud Foundation software stack. Registry for storing, managing, and securing Docker images. Container environment security for each stage of the life cycle. Solution for running build steps in a Docker container.
Containers with data science frameworks, libraries, and tools. Containerized apps with prebuilt deployment and unified billing. Package manager for build artifacts and dependencies. Components to create Kubernetes-native cloud-based software. IDE support to write, run, and debug Kubernetes applications. Platform for BI, data applications, and embedded analytics. Messaging service for event ingestion and delivery.
Service for running Apache Spark and Apache Hadoop clusters. Data integration for building and managing data pipelines. Workflow orchestration service built on Apache Airflow. Service to prepare data for analysis and machine learning. Intelligent data fabric for unifying data management across silos. Metadata service for discovering, understanding, and managing data.
Service for securely and efficiently exchanging data analytics assets. Cloud-native wide-column database for large scale, low-latency workloads. Cloud-native document database for building rich mobile, web, and IoT apps. In-memory database for managed Redis and Memcached. Cloud-native relational database with unlimited scale and Serverless, minimal downtime migrations to Cloud SQL. Infrastructure to run specialized Oracle workloads on Google Cloud.
NoSQL database for storing and syncing data in real time. Serverless change data capture and replication service. Universal package manager for build artifacts and dependencies. Continuous integration and continuous delivery platform.
Service for creating and managing Google Cloud resources. Command line tools and libraries for Google Cloud. Cron job scheduler for task automation and management. Private Git repository to store, manage, and track code. Task management service for asynchronous task execution. Fully managed continuous delivery to Google Kubernetes Engine. Full cloud control from Windows PowerShell. Healthcare and Life Sciences. Solution for bridging existing care systems and apps on Google Cloud.
Tools for managing, processing, and transforming biomedical data. Real-time insights from unstructured medical text. Integration that provides a serverless development platform on GKE. Tool to move workloads and existing applications to GKE. Service for executing builds on Google Cloud infrastructure. Traffic control pane and management for open service mesh. API management, development, and security platform.
Fully managed solutions for the edge and data centers. Internet of Things. IoT device management, integration, and connection service. Automate policy and security for your deployments. Dashboard to view and export Google Cloud carbon emissions reports.
Programmatic interfaces for Google Cloud services. Web-based interface for managing and monitoring cloud apps. App to manage Google Cloud services from your mobile device. Interactive shell environment with a built-in command line. Kubernetes add-on for managing Google Cloud resources. Tools for monitoring, controlling, and optimizing your costs. Tools for easily managing performance, security, and cost. Service catalog for admins managing internal enterprise solutions.
Open source tool to provision Google Cloud resources with declarative configuration files. Media and Gaming. Game server management service running on Google Kubernetes Engine. Open source render manager for visual effects and animation. Convert video files and package them for optimized delivery. App migration to the cloud for low-cost refresh cycles. Data import service for scheduling and moving data into BigQuery.
Reference templates for Deployment Manager and Terraform. Components for migrating VMs and physical servers to Compute Engine. Storage server for moving large volumes of data to Google Cloud. Data transfers from online and on-premises sources to Cloud Storage. Migrate and run your VMware workloads natively on Google Cloud. Security policies and defense against web and DDoS attacks.
Content delivery network for serving web and video content. Domain name system for reliable and low-latency name lookups. Service for distributing traffic across applications and regions. NAT service for giving private instances internet access. Connectivity options for VPN, peering, and enterprise needs. Connectivity management to help simplify and scale networks.
Network monitoring, verification, and optimization platform. Cloud network options based on performance, availability, and cost. VPC flow logs for network monitoring, forensics, and security. Google Cloud audit, platform, and application logs management.
Infrastructure and application health with rich metrics. Application error identification and analysis. GKE app development and troubleshooting. Tracing system collecting latency data from applications. CPU and heap profiler for analyzing application performance. Real-time application state inspection and in-production debugging. Tools for easily optimizing performance, security, and cost. Permissions management system for Google Cloud resources. Compliance and security controls for sensitive workloads.
Manage encryption keys on Google Cloud. Encrypt data in use with Confidential VMs. Platform for defending against threats to your Google Cloud assets. Sensitive data inspection, classification, and redaction platform. Managed Service for Microsoft Active Directory. Cloud provider visibility through near real-time logs.
Two-factor authentication device for user account protection. Store API keys, passwords, certificates, and other sensitive data. Zero trust solution for secure application and resource access. Platform for creating functions that respond to cloud events. Workflow orchestration for serverless products and API services. Cloud-based storage services for your business.
File storage that is highly scalable and secure. Block storage for virtual machine instances running on Google Cloud. Object storage for storing and serving user-generated content. Block storage that is locally attached for high-performance needs. Contact us today to get a quote. Request a quote. Google Cloud Pricing overview. Pay only for what you use with no lock-in. Get pricing details for individual products.
Related Products Google Workspace. Get started for free. Self-service Resources Get started. Stay in the know and become an Innovator. Prepare and register for certifications. Expert help and training Consulting. Partner with our experts on cloud projects. Enroll in on-demand or classroom training. Partners and third-party tools Google Cloud partners. Explore benefits of working with a partner.
Join the Partner Advantage program. Deploy ready-to-go solutions in a few clicks. Machine type families. Regions and zones. Get started. Plan and prepare. Work with regions and zones. Images and operating systems. OS images. Premium operating systems. Access control. Create VMs. Create a VM. Create Spot VMs. Spot VMs. Preemptible VMs. Create custom images. Create and manage instance templates. Create multiple VMs. Create a managed instance group MIG.
Create sole-tenant VMs. Use nested virtualization. Manage VM boot disks. Migrate VMs. Import disks and images. Automatic import. Manual import. Move a VM within Google Cloud. Connect to VMs. Connection methods. Access management. IAM-based access control. SSH key management. Best practices. Manage storage.
About disks. Disk encryption and security. Persistent disks. Manage disk performance. Use regional persistent disks for high availability services. Ephemeral disks local SSD. Manage local SSD performance. Back up and restore. Back up VMs. Back up disks. Create application consistent snapshots. Restore from a backup. Manage VMs. Basic operations and lifecycle. Stop and start VMs.
View VM properties. Update VM details. Configure IP addresses. Delete VMs.
Question manageengine free edition absolutely
Следующая статья vnc windows server 2008 64 bits