今日のつぶやき

Twilog

キーワードはモバイル、ソーシャル、リアルタイム。ビジネス・モデル、ビジネス・システムにどのような影響があるか。というよりどのような再定義があるか。再定義された枠組みの中でのモデリングの位置付けについて考えてみたい。 #

今回の #pdc09 はクラウド・プラットフォームであるWindow Azureが議題の中心になりそうだけど、そのプラットフォームの上でモバイル、ソーシャル、リアルタイムをどのように実現していこうとしているのかの兆しに着目している。 #

[Twitter]今日のつぶやき ff.im/bxu0O #

スーパーで買ってきたStarbucksのfrappuccinoはコーヒー牛乳の味。 #

Red Bull初めて飲んでみた。リポビタンDが大発明だったということだなぁ。 #

#pdc09 のセッションルームに到着。無線LANがつながった。 #

『Architecting and Developing for the Windows Azure Platform』 #

RESTでどこまでできるかなー。チェックしたいポイント。 #pdc09 #

『Introduction. Intro to example app』 #pdc09 #

TicketDirect : example application #pdc09 #

デモ動かない #

画面はAjax使ったMac風のWeb #

立ち上げなおしたら動いた。 #pdc09 #

Azure Roles, Azure Storages, SQL Azure, Client Applications #pdc09 #

Sofare as a Service, White-label VARs/ISVs, User/Customers/Consumers #pdc09 #

Platform as a ServiceまでWindows Azureはカバーしている。Amazon EC2はInfrastructre as a Serviceまで。 #pdc09 #

Salesforce.com, Microsoft CRM LiveはSoftware as a Serviceの半分までカバー。 #pdc09 #

Market Quadrant, Pay as you go/Buy up-front, Platform niche player/Vertially Integrated #pdc09 #

Microsoft, Google SalesforceはPay as yo go + Vertically Integrated。 #pdc09 #

Amazonはpay as you go #pdc09 #

Pay as yo goのcashflowは平準化する。 #pc09 #

#pdc09 Consolication Cloud, Scale Out Cloud #

#pdc09 Reliability through Software #

#pdc09 High Scale Application Archetype #

#pdc09 じPeriodicity of Deman、ピーク時の不足、閑散期の無駄 #

#pdc09 forward processing, stupidなリクエストをクラウドで処理してon primseに送る #

#pdc09 Sisyphean Task データセンターの維持 #

#pdc09 Data Sovereignty and Security #

#pdc09 Service Level Agreements #

#pdc09 セキュリティとかSLAの問題をどのようにして受け入れてもらえるようにするのか。問題提起ということかな。 #

#pdc09 『Scalability, Caching and Elasticity』 #

#pdc09 vertical scale up, horizontal scale out #

#pdc09 for small scenarios scale up is cheaper. #

#pdc09 For larger scenarios scale out only solution. #

#pdc09 Scald out offers promise of linear, infinite scale. #

#pdc09 scale outがonly solutionという認識が大事。 #

#pdc09 Scalability != Performance #

#pdc09 リアルタイムが早いと勘違いしているケースもある。スケーラビリティも早いと勘違いするケースもありそう。 #

#pdc09 Achieving Linear Scale Out。リニアにスケールさせることが大事。 #

#pdc09 Reduce or eliminate shared resources #

#pdc09 homogenous, stateless computation nodes #

#pdc09 Azureは最低限動作するインスタンス数を*.cscfgで指定できる。(appengineはアクセスがないと全て落ちる→spin off問題) #

#pdc09 インスタンスを立ち上げっぱなしにすると相応のコストがかかるはず。課金すればよいけど。 #

#pdc09 Small 1, Medium 2, Large 4, Extra Large 8 #

#pdc09 Small 1, 1.7GB, 250GB/ Medium 2, 3.5GB, 500GB/ Large 4, 7GB, 100GB/Extra large 8, 15gb, 2000gb #

#pdc09 VMサイズを選べるのはapengineにない特長。 #azure #

#pdc09 デモしてる人のブラウザにgoogle tool barが。 #

#pdc09 マルチコアで高速に動かないアプリはクラウドでも高速に動かない。 #

#pdc09 ParellForeach #

#pdc09 Caching #

#pdc09 cachingがクラウドの鍵となる技術。 Few Azure applications can do without caching. #

#pdc09 cachi can improve both performance and scalability. #

#pdc09 caching strategies/ client side caching, static content genreration, output caching, fragment cdachieng, +1 #

#pdc09 client caching - ETags, soft caching, header added on http response #

#pdc09 ETag problems: stil requires round trip to server. may require execution of server side code to recreate ETag befre checking. #

#pdc09 clinet cacing - Cache-Control #

#pdc09 cache-control, hard caching, header added on http rsponse, client may cache file without further request for 30 days #

#pdc09 very useful for static files, no way to implement in Blob storage #

#pdc09 cache-control problems, no blob sorage support, what if files do change in the 30 days? #

#pdc09 benfits prevents un-necessary http requests, prevents un-necessary donloads #

#pdc09 azure technique: put static files in web role, use cache-control + url flipping #

#pdc09 static content generatin #

#pdc09 generate content periodically in worker role #

#pdc09 content may be full pages, resource, content fragments #

#pdc09 push static content into Blog storage #

#pdc09 serve direct out of blob storage, retrieve via a web role, may be local database ... #

#pdc09 problems - need to deal with stale data. #

#pdc09 benefits - reduce load on web roles, potentially reduce load on data tier, response times improved, #

#pdc09 can bombine with cache-control and etags #

#pdc09 Output Caching #

#pdc09 ASP.NET feature #

#pdc09 problems - mutiplie copies fof cached data. 1/instance, Still requires client roud trip each request, #

#pdc09 may beed URL flipping to refresh client cache. #

#pdc09 Fragment Caching #

#pdc09 problems - multiple copies of dcached data. 1/instance, #

#pdc09 still requires client round trip each request, + 1 #

#pdc09 Data Caching #

#pdc09 ASP.NET Cache/ Memcached, Velocity, Shared Cache/Cloud arguably 'cache' in Blob storage. #

#pdc09 most flexible - most code to write #

#pdc09 specialized cache servers vs caching on all roles #

#pdc09 problems basic asp.net cache suffers consistency issues #

#pdc09 3rd party cache tools require work for azure support. #

#pdc09 stale data still an issue. #

#pdc09 benefis - very flexible, reduces load on data tier, most efficient use of memory, any type of data. #

#pdc09 Web Roles, WCF roles, dynamic worker, distributed cache worker, partitioner worker --- デモのazure role #

#pdc09 3rd party cache frameworkを使っている。 #

web and wcf roles share common cache. #

#pdc09 session state (our own impl) is stored in cache. #

#pdc09 『Elastic Scale Out』 #

#pdc09 application load almost always varies #

#pdc09 reqular and predictable, irregular an dpredictable, unpredictable #

#pdc09 maintaining excess capacity or headroom, adding/removing additional capacity --- Dealing with variable load takes two forms #

#pdc09 head room in azure platform services - web roles, run additonal web roles, handle additional laod before performance degrages #

#pdc09 worker roles - if possible just buffer into queues, will be driven by tolerable level of latency, #

#pdc09 start addional role sonly if queues not cleanring. #

#pdc09 Azure Storage - Host nodes *may* have headroom. 3 replicas give storage fabric options, opqeue to the azure customer. #

#pdc09 SQL Azure - Non-deterministic throttle gives little indication, run extra instances #

#pdc09 web roles/worker roles - enable more instances, editing instance count in config leaves existing instances running #

#pac09 change to using larger VMs- will require redeploy. Need to see pricding. #

#pdc09 Azure Storage - opqeue to use, partition agressivly , #

#pdc09 could 'heat up' a partition to encourage scale up - constly due to transaction costs. #

#pdc09 SQL Azure - add more databases, very difficult to achieve mid-stream, #

#pdc09 requires moving hot data, maintaining consistency across multiple dbs, will *** #

#pdc09 Rule Based Scaling #

#pdc09 use service management api #

pdc09 predicable or periodic demand, unpredictable demand #

#pdc09 Monitor metrics #

#pdc09 requests per second, queue messages processed/ interval --- primary metrics #

#pdc09 cpu utilization, queue length, response time -- secandary metrics #

#pdc09 gathering metrics #

#pdc09 capture various metrics - azure logs, performance counters, IIS logs, etc... --- capture varios metrics #

#pdc09 Evaluating business rules #

#pdc09 requests, jobs in my queue, money this month? #

#pdc09 take action -- add/remove instances, change role size, send notifications #

#pdc09 情報を集める方法、集めた情報からスケジュールする方法 #

#pdc09 Dynamic Scaling Engine #

#pdc09 summary -- designing for multiple instances provides scale out, availability, elasticity options. #

#psc09 cacing should be a key component of any azure application. #

昼食は、チーズバーガピザ。 #

#pdc09 午前のScalability an dElasticityのsummaryの最後の項目が抜けてた。多分、Scalabilityの管理、自動化だと思う。 #

#pdc09 午後の最初『Asyncrouns Work』 #

#pdc09 improve apparent responsiveness. #

#pdc09 establish mutiple units of scale. #

#pdc09 basic asyncronous patterns #

#pdc09 Web RoleがWorker RoleにQueueで仕事をリクエストする。 #

#pdc09 アップロードされたイメージをBlobに入れて、Blog　URIとメタデータをqueue経由でworker roleに渡す。 #

#pdc09 soft transaction concept. #

#pdc09 worker roleはサムネールを生成してtableに入れて、キューの内容を消す。 #

#pdc09 idempotency f(x) = f(f(x)) #

#pdc09 message process At Least Once。 #

#pdc09 idempotencyとかAt Least Once懐かしー #

#pdc09 solving the idempotency problem #

#pdc09 Transaction IDを比較してチェックする。 #

#pdc09 crud - readとdeleteはidempotent。createは違う。updateは... #

#pdc09 idempotencyのある処理と違う処理を区別する、idempotencyのない処理はtransaction idをチェックして対処する、ということになるのかな。これから先のストーリ。 #

#pdc09 Replay Log Storage #

#pdc09 Poison Message Handling #

#pdc09 poison messages may become zombies #

#pdc09 even idempotent messages can be poisonous #

#pdc09 spitting the poison pill - queueを使う。message id GUID #

#pdc09 a guid created by queue service on enqueue immutable and not user defined. #

#pdc09 insert/update ID + Count in storage. #

#pdc09 keep a count of the number of times a message has been processed. #

#pdc09 set threasholds before a message is placed into a Poison queue and delted. #

#pdc09 ensure that your poison test is at the top of the batch. +1 #

#pdc09 Replay Log vs Poison Tracker #

#pdc09 run the risk of wroker crashing before poison check. #

#pdc09 Dynamic Worker Role #

#pdc09 Dynamic Worker Roles #

#pdc09 worker roles most cost effective at 100% utilization #

#pdc09 often have many work types, none of which requires 100% of 1 instance #

want to add new work types without redeploying #

#pdc09 solution: Use a generic queue. #

pdc09 encode message with info to resolve work. type laod assembly to process mesage from blob storage. dynamically instantiate and execute #

#pdc09 tinyurl.com/lokad-cloud #

#pdc09 demoでOpenIdログインした。 #

#pdc09 use a smart polling approach for queues #

#pdc09 use a new AppDomain to separete loaded types #

#pdc09 may be value in including Replay Log and Poison tracking as part of Dynamic Worker framework. #

#pdc09 The MapReduce Pattern #

#pdc09 very paralelizable - less efficient. -- Trading performance for increased scale again. #

#pdc09 Specific impl. of dynamics workers. #

#pdc09 Map function, grouping, redeuce function #

#pdc09 generally processor intensive/RAM light - do worker rokes have too much RAM? #

#pdc09 No out of the box support yet. - see lokad demo shortly #

#pdc09 still limitations in Azure with Colocation of Data and Computation #

#pdc09 dataとcomputationのcolocationができないのが問題。Azureの新しいサービスの前振りかも。 #

#pdc09 このデモではMapReduceを自作してるけど、Azureの新機能でMapReduceが出てくる前振りかもしれない。 #

#pdc09 asynchronous processing can massively increase the perceived speed of your application #

#pdc09 dispatch via azure queues requirees some case. - ensure that your messages are idempotennt or have a commponsation. ... #

#pdc09 『Partitioninig Strategy』 #

#pdc09 Horizontal partitioninig #

#pdc09 Spread Data Across Similar Nodes #

#pdc09 Archieve Massive Scale Out (Data and Load) #

#pdc09 Intra-Partition Queries Simple #

#pdc09 Cross-Partition Queries Harder #

#pdc09 Vertical Partitioning #

#pdc09 Spread Data Across dis-Similar Nodes #

#pdc09 Place frequently queried data in 'expensive' indexed storage #

#pdc09 place large data in 'cheap' binary storage #

#pdc09 retrieving a whole row requires > 1 query #

#pdc09 Hybrid partitioning #

#pdc09 Horizontal partitioning - azure tables #

#pdc09 Azure Table Storage #

#pdc auto-balanced, hot partitions meay be scaled up, partition key AND row key = primary key #

#pdc09 continuation tokens may be returned from cross partition queries #

#pdc09 key columns up to 1kb in size #

#pdc09 be aggressive #

#pdc09 SQL Azure - Key Pointsl..続き中 #

#pdc09 partition for data volumen > 10GB #

#pdc09 all partition logic up to teh developer -- algorithmic, lookup based #

#pdc09 partitions are not Auto-Balanced #

#pdc09 (transaction throttle (non determinisstic) Always code for retry #

#pdc09 choosing a partition key -- natural keys, mathematical, ... #

#pdc09 Using Modulo #

#pdc09 Using Hash Values (partitionining) #

#pdc09 do not use a cryptographic hash function #

#pdc09 tinyurl.com/part-hash #

#pdc09 be careful if using Object.GetHashCode() #

#pdc09 partition stability over time #

#pdc09 may need pto change artitionning scheme #

#pdc09 re-partition all data or; version partitioning scheme #

#pdc09 v1 = GUID mod 4
v2 = GUID mod 10 #

#pdc09 Just in time partitionning #

#pdc09 in sql azure partitions cost money, In highly elastic scenarios partitions may be needed for just a few hours or days, #

#pdc09 If load is predictable - partition before load commences, de-partition after load has subsides. #

#pdc09 goals for vertical partitioning #

#pdc09 balance performance vs cost #

#pdc09 use appropriate storage for type of data #

#psc09 sql azure - fuly indexable, noquery transaxtion charge, $9.99/gb #

#pdc09 azure storage - limited indexed ... #

#pdc09 vertical partitioning example #

#pdc09 searchable data in azure tables or SQL azure - indexed, no cost per query #

#pdc09 thumbnails in azure tables #

#pdc09 binary properties < 64Kb #

#pdc09 batch queries saves transaction costs, - duplicated for EACH user #

#pdc09 full photos in azure blob storage - can handle larege data - can stream full sized files direct... #

#pdc09 Cost per month = .0026 cents #

#pdc09 de-normalized - cost per month = .0017 cents #

#pdc09 『NoSQL/Non-Relational Data Modeling』 #

#pdc09 Azure Tables != RDBMS, Storage is cheap, cross partition queries are resource intensive, de-normalization foten name of the game #

#pdc09 tweetからworker roleがtweetindexを生成。ここに全てが入っている。with azure tables we go the whole way. #

#pdc09 TweetIndexでなくてmentionIndexでした。 #

#pdc09 And then we start going crazy. #

#pdc09 just in time partitionninig is everything. #

#pdc09 we partion just for peak sales period. #

#pdc09 partition our seating tables into new databases #

#pdc09 summary - partitioning data key to cloud scale apps #

#pdc09 horizontallly partition for scale out #

#pdc09 verically partition for cost/performance #

#pdc09 azure storage requires different approach to data modeling #

#pdc09 don't be afraid to aggressively de-normalize and duplicate data. #

#pdc09 モデリングはさわりぐらいだった。 #

#pdc09 『pricing and Business Model』 #

#pdc09 プライスの話かーと思ったけど、アプリケーション・アーキテクチャによって運用コストが変わってくるので、テクノロジー的にも大事。 #

#pdc09 data within datacenter is unmetered. Bwtween datacenter is metered. #

#pdc09 12c per hour for each deployed instance. #

空うさぎのバージョンを上げてみた。 #

#pdc09 workload resource usage -- storage txn, data in/out #

#pdc09 static resource usage -- bytes on disk, requireed sql azure instances #

#pdc09 required instances -- based on simple average requests/second or; more sophisticated model #

#pdc09 add instances to make numbers 'reasonable'. #

#pdc09 compress blob data on disk -- set content-encoding header when putting blob #

#pdc09 use caching to reduce request txn count #

#pdc09 use css sprites and data uris to rereduce txn cout #

#pdc09 batch your requests #

#pdc09 in the web role -- incurs additional 'free' but resource consuming requests to retrieve #

#pdc09 In blob storage --　... #

#pdc09 sql azure - vertically partition large data columns out to Azure Storage #

#pdc09 support dynamic partitioning if possible #

#pdc09 consider just-in-time partitioning #

#pdc09 pull archive data out of the cloud to cheaper on premise storage or to Azure storage. #

#pdc09 『Diagnostics and Deployment』 #

#pdc09 fabricにアプリケーションのメタ情報を定義するのかな。 #

#pdc09 diagnosticsを取ってくるデモしてる。 #

dev fabric prevents off machine calls #

#pdc09 to resolve. reverse proxy on local machine. - use internet redirection server (rinetd). - use fiddler #

#pdc09 reduces security risk footprint, makes running test/integration servers hard ← dev fabric prevents off machine calss #

#pdc09 fault domains, upgrade domains #

#pdc09 運用システムと開発中システムの2つを用意して、インスタンスをスワップしてシステムを入れ替えることができる、らしい。VIP Swap Upgrade #

#pdc09 for the best user experience. Invest in warming up instances in staging before swapping. #

#pdc09 In-place Upgrade #

#pdc09 fabricのXMLをいじって、upgradeのstrategyを変えている。デモ。 #

#pdc09 In-place upgrade -- only 2 upgrade domains. - instances torn down before new instances brought up #

#pdc09 geo-location & affinity groups #

#pdc09 affinitized - groups services with dependent resources - ensures close geolocation #

#pdc09 - un-affinitized - can still set explicit geo region - may not be guaranteed #

#pdc09 Programmatic Deployment #

#pdc09 Automating Deployment #

おわりー #

ホテルの部屋で一服。帰ってきたらりんごとパプリカとひょうたんが成っていてうれしい。(サンシャイン牧場) #

#pdc09 AzureはAPIが出ていて相当チューニングできそう。AppEngine的に全てお任せもできると思われるので、お任せ運用からバリバリ運用まで色々なニーズに対応できそうな印象を持った。 #

Automatically shipped by LoudTwitter